2023-07-20T17:17:38.043 INFO:root:teuthology version: 0.0.1.dev153+g79491cb 2023-07-20T17:17:38.044 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} --archive /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337 --verbose --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/orig.config.yaml 2023-07-20T17:17:38.100 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T17:17:38.197 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337 branch: wip-pdonnell-testing-20230719.201220 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7345337' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230719.201220 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230719.201220 sha1: 78296bf9274b016ce68ff036026e51a56af22f7c owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8388 sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 sleep_before_teardown: 0 subset: 12/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230719.201220 suite_path: /home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 78296bf9274b016ce68ff036026e51a56af22f7c syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi049.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi079.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 79491cb69dd37a1eafb93b3471998174bdfa2fd9 timestamp: 2023-07-20_16:47:59 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-20T17:17:38.197 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa; will attempt to use it 2023-07-20T17:17:38.198 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks 2023-07-20T17:17:38.198 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-20T17:17:38.204 INFO:teuthology.task.internal:Checking packages... 2023-07-20T17:17:38.232 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '1b0b2eb005a7c51d97a788b9900d511e1394b537' 2023-07-20T17:17:38.232 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-20T17:17:38.233 INFO:teuthology.packaging:ref: None 2023-07-20T17:17:38.233 INFO:teuthology.packaging:tag: None 2023-07-20T17:17:38.233 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230719.201220 2023-07-20T17:17:38.233 INFO:teuthology.packaging:sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T17:17:38.234 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230719.201220 2023-07-20T17:17:38.590 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5040.g1b0b2eb0 2023-07-20T17:17:38.591 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-20T17:17:38.599 INFO:teuthology.task.internal:no buildpackages task found 2023-07-20T17:17:38.599 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-20T17:17:38.624 INFO:teuthology.task.internal:Saving configuration 2023-07-20T17:17:38.650 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-20T17:17:38.659 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T17:17:38.688 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 17:11:23.773279', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T17:17:38.713 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 17:11:23.769981', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T17:17:38.741 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 17:11:23.772085', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T17:17:38.741 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-20T17:17:38.748 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-20T17:17:38.749 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-20T17:17:38.749 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-20T17:17:38.749 INFO:teuthology.run_tasks:Running task console_log... 2023-07-20T17:17:38.868 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3fa3829430>, signals=[15]) 2023-07-20T17:17:38.872 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-20T17:17:38.879 INFO:teuthology.task.internal:Opening connections... 2023-07-20T17:17:38.879 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2023-07-20T17:17:38.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T17:17:38.952 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2023-07-20T17:17:38.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T17:17:39.021 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2023-07-20T17:17:39.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T17:17:39.089 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-20T17:17:39.096 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2023-07-20T17:17:39.117 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2023-07-20T17:17:39.118 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2023-07-20T17:17:39.177 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T17:17:39.178 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T17:17:39.178 INFO:teuthology.orchestra.run.smithi049.stdout:ID="rhel" 2023-07-20T17:17:39.178 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="fedora" 2023-07-20T17:17:39.178 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="8.6" 2023-07-20T17:17:39.178 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el8" 2023-07-20T17:17:39.178 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T17:17:39.179 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2023-07-20T17:17:39.179 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T17:17:39.179 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T17:17:39.179 INFO:teuthology.orchestra.run.smithi049.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T17:17:39.179 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T17:17:39.179 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:17:39.179 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T17:17:39.179 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T17:17:39.180 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T17:17:39.180 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T17:17:39.180 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2023-07-20T17:17:39.199 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2023-07-20T17:17:39.219 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2023-07-20T17:17:39.219 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2023-07-20T17:17:39.278 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T17:17:39.278 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T17:17:39.279 INFO:teuthology.orchestra.run.smithi079.stdout:ID="rhel" 2023-07-20T17:17:39.279 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="fedora" 2023-07-20T17:17:39.279 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="8.6" 2023-07-20T17:17:39.279 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el8" 2023-07-20T17:17:39.279 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T17:17:39.279 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2023-07-20T17:17:39.279 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T17:17:39.279 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T17:17:39.280 INFO:teuthology.orchestra.run.smithi079.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T17:17:39.280 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T17:17:39.280 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:17:39.280 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T17:17:39.280 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T17:17:39.280 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T17:17:39.280 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T17:17:39.281 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2023-07-20T17:17:39.307 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2023-07-20T17:17:39.328 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2023-07-20T17:17:39.328 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2023-07-20T17:17:39.387 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T17:17:39.387 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T17:17:39.387 INFO:teuthology.orchestra.run.smithi097.stdout:ID="rhel" 2023-07-20T17:17:39.387 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="fedora" 2023-07-20T17:17:39.388 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8.6" 2023-07-20T17:17:39.388 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8" 2023-07-20T17:17:39.388 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T17:17:39.388 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2023-07-20T17:17:39.388 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T17:17:39.388 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T17:17:39.388 INFO:teuthology.orchestra.run.smithi097.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T17:17:39.389 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T17:17:39.389 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:17:39.389 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T17:17:39.389 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T17:17:39.389 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T17:17:39.389 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T17:17:39.390 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-07-20T17:17:39.415 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-20T17:17:39.427 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-20T17:17:39.436 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-20T17:17:39.436 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2023-07-20T17:17:39.439 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2023-07-20T17:17:39.441 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2023-07-20T17:17:39.458 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-20T17:17:39.466 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-20T17:17:39.467 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2023-07-20T17:17:39.498 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2023-07-20T17:17:39.500 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2023-07-20T17:17:39.535 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-20T17:17:39.620 INFO:teuthology.run_tasks:Running task kernel... 2023-07-20T17:17:39.636 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-07-20T17:17:39.637 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-07-20T17:17:39.637 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-07-20T17:17:39.637 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T17:17:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:17:39.657 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2023-07-20T17:17:39.716 INFO:teuthology.orchestra.run.smithi049.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-20T17:17:39.717 INFO:teuthology.task.kernel:Running kernel on smithi049: 4.18.0-372.9.1.el8.x86_64 2023-07-20T17:17:39.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2023-07-20T17:17:40.807 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-07-20T17:17:40.807 INFO:teuthology.orchestra.run.smithi049.stdout:Unable to read consumer identity 2023-07-20T17:17:42.786 INFO:teuthology.orchestra.run.smithi049.stdout:Copr repo for python3-asyncssh owned by ceph 4.8 kB/s | 3.5 kB 00:00 2023-07-20T17:17:46.295 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 16 MB 00:03 2023-07-20T17:17:49.878 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 641 kB/s | 24 kB 00:00 2023-07-20T17:17:50.747 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-20T17:17:50.803 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-07-20T17:17:50.804 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-07-20T17:17:50.804 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-07-20T17:17:50.930 DEBUG:teuthology.orchestra.run.smithi049:> echo no | sudo yum reinstall kernel || true 2023-07-20T17:17:51.281 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-07-20T17:17:51.282 INFO:teuthology.orchestra.run.smithi049.stdout:Unable to read consumer identity 2023-07-20T17:17:51.566 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:02 ago on Thu 20 Jul 2023 05:17:49 PM UTC. 2023-07-20T17:17:51.611 INFO:teuthology.orchestra.run.smithi049.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T17:17:51.646 INFO:teuthology.orchestra.run.smithi049.stderr:Error: No packages marked for reinstall. 2023-07-20T17:17:51.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum reinstall -y kernel || true 2023-07-20T17:17:52.041 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-07-20T17:17:52.042 INFO:teuthology.orchestra.run.smithi049.stdout:Unable to read consumer identity 2023-07-20T17:17:52.316 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:03 ago on Thu 20 Jul 2023 05:17:49 PM UTC. 2023-07-20T17:17:52.357 INFO:teuthology.orchestra.run.smithi049.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T17:17:52.386 INFO:teuthology.orchestra.run.smithi049.stderr:Error: No packages marked for reinstall. 2023-07-20T17:17:52.439 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2023-07-20T17:17:52.522 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-20T17:17:52.522 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-20T17:17:52.522 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-20T17:17:52.523 DEBUG:teuthology.parallel:result is None 2023-07-20T17:17:52.523 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-20T17:17:52.531 INFO:teuthology.task.internal:Creating test directory... 2023-07-20T17:17:52.531 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T17:17:52.533 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T17:17:52.536 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T17:17:52.557 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-20T17:17:52.565 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-20T17:17:52.576 INFO:teuthology.task.internal:Creating archive directory... 2023-07-20T17:17:52.577 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T17:17:52.590 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T17:17:52.596 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T17:17:52.631 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-20T17:17:52.640 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-20T17:17:52.640 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T17:17:52.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:17:52.682 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T17:17:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:17:52.702 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T17:17:52.704 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T17:17:52.725 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T17:17:52.747 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T17:17:52.760 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T17:17:52.763 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T17:17:52.779 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T17:17:52.788 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T17:17:52.806 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T17:17:52.807 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-20T17:17:52.816 INFO:teuthology.task.internal:Configuring sudo... 2023-07-20T17:17:52.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T17:17:52.819 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T17:17:52.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T17:17:52.852 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-20T17:17:52.863 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-20T17:17:52.864 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T17:17:52.886 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T17:17:52.893 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T17:17:52.911 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T17:17:52.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T17:17:53.031 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T17:17:53.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T17:17:53.158 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:17:53.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T17:17:53.225 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T17:17:53.275 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T17:17:53.330 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T17:17:53.418 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T17:17:53.460 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-07-20T17:17:53.461 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T17:17:53.534 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T17:17:53.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T17:17:53.641 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T17:17:53.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T17:17:53.731 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-07-20T17:17:53.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T17:17:53.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2023-07-20T17:17:53.813 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2023-07-20T17:17:53.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2023-07-20T17:17:53.876 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T17:17:53.888 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T17:17:53.915 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T17:17:54.324 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-20T17:17:54.339 INFO:teuthology.task.internal:Starting timer... 2023-07-20T17:17:54.340 INFO:teuthology.run_tasks:Running task pcp... 2023-07-20T17:17:54.352 INFO:teuthology.run_tasks:Running task selinux... 2023-07-20T17:17:54.368 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-07-20T17:17:54.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2023-07-20T17:17:54.438 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2023-07-20T17:17:54.440 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2023-07-20T17:17:54.502 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2023-07-20T17:17:54.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2023-07-20T17:17:54.561 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2023-07-20T17:17:54.563 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-20T17:17:54.563 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2023-07-20T17:17:54.593 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2023-07-20T17:17:54.593 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2023-07-20T17:17:54.621 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2023-07-20T17:17:54.621 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2023-07-20T17:17:54.642 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2023-07-20T17:17:54.642 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi049.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi097.front.sepia.ceph.com': 'permissive'} 2023-07-20T17:17:54.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T17:17:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:17:54.673 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T17:17:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:17:54.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T17:17:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:17:54.730 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-20T17:17:54.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2023-07-20T17:17:54.758 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2023-07-20T17:17:54.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2023-07-20T17:17:54.821 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-20T17:17:54.832 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-07-20T17:17:54.833 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-20T17:17:54.856 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-20T17:17:54.857 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com,smithi079.front.sepia.ceph.com,smithi097.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-20T17:29:32.194 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com'), Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi097.front.sepia.ceph.com')] 2023-07-20T17:29:32.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-07-20T17:29:32.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T17:29:32.267 DEBUG:teuthology.orchestra.run.smithi049:> true 2023-07-20T17:29:32.334 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-07-20T17:29:32.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2023-07-20T17:29:32.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T17:29:32.401 DEBUG:teuthology.orchestra.run.smithi079:> true 2023-07-20T17:29:32.471 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2023-07-20T17:29:32.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-07-20T17:29:32.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T17:29:32.540 DEBUG:teuthology.orchestra.run.smithi097:> true 2023-07-20T17:29:32.566 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-07-20T17:29:32.567 INFO:teuthology.run_tasks:Running task clock... 2023-07-20T17:29:32.581 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-20T17:29:32.581 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T17:29:32.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T17:29:32.585 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T17:29:32.586 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T17:29:32.588 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T17:29:32.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T17:29:32.617 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T17:29:32.619 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T17:29:32.634 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T17:29:32.634 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T17:29:32.640 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T17:29:32.654 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T17:29:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2023-07-20T17:29:32.715 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2023-07-20T17:29:32.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2023-07-20T17:29:32.722 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2023-07-20T17:29:32.727 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2023-07-20T17:29:32.734 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2023-07-20T17:29:32.738 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T17:29:32.742 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T17:29:32.748 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T17:29:32.752 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T17:29:32.756 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T17:29:32.762 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T17:29:32.821 INFO:teuthology.orchestra.run.smithi049.stderr:bash: ntpq: command not found 2023-07-20T17:29:32.825 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T17:29:32.825 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-07-20T17:29:32.826 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.826 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.826 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.826 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.830 INFO:teuthology.orchestra.run.smithi097.stderr:bash: ntpq: command not found 2023-07-20T17:29:32.834 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T17:29:32.834 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2023-07-20T17:29:32.834 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.834 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.834 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.835 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.846 INFO:teuthology.orchestra.run.smithi079.stderr:bash: ntpq: command not found 2023-07-20T17:29:32.850 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T17:29:32.850 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2023-07-20T17:29:32.850 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.851 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.851 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.851 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:29:32.852 INFO:teuthology.run_tasks:Running task pexec... 2023-07-20T17:29:32.865 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-20T17:29:32.866 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi049.front.sepia.ceph.com 2023-07-20T17:29:32.866 DEBUG:teuthology.orchestra.run.smithi049:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T17:29:32.867 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi079.front.sepia.ceph.com 2023-07-20T17:29:32.867 DEBUG:teuthology.orchestra.run.smithi079:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T17:29:32.868 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi097.front.sepia.ceph.com 2023-07-20T17:29:32.868 DEBUG:teuthology.orchestra.run.smithi097:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T17:29:33.247 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-07-20T17:29:33.259 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-07-20T17:29:33.264 INFO:teuthology.orchestra.run.smithi079.stdout:Updating Subscription Management repositories. 2023-07-20T17:29:34.394 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:04:21 ago on Thu 20 Jul 2023 05:25:12 PM UTC. 2023-07-20T17:29:34.430 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:39 ago on Thu 20 Jul 2023 05:27:54 PM UTC. 2023-07-20T17:29:34.446 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:34 ago on Thu 20 Jul 2023 05:25:59 PM UTC. 2023-07-20T17:29:36.155 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-07-20T17:29:36.156 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-07-20T17:29:36.156 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2023-07-20T17:29:36.156 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-07-20T17:29:36.156 INFO:teuthology.orchestra.run.smithi079.stdout:Resetting modules: 2023-07-20T17:29:36.156 INFO:teuthology.orchestra.run.smithi079.stdout: container-tools 2023-07-20T17:29:36.157 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:29:36.157 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-07-20T17:29:36.157 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-07-20T17:29:36.157 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:29:36.157 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-07-20T17:29:36.158 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-07-20T17:29:36.158 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2023-07-20T17:29:36.158 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-07-20T17:29:36.158 INFO:teuthology.orchestra.run.smithi097.stdout:Resetting modules: 2023-07-20T17:29:36.158 INFO:teuthology.orchestra.run.smithi097.stdout: container-tools 2023-07-20T17:29:36.158 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:29:36.158 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-07-20T17:29:36.159 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-07-20T17:29:36.159 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:29:36.427 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-07-20T17:29:36.429 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-07-20T17:29:36.429 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-07-20T17:29:36.429 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-07-20T17:29:36.429 INFO:teuthology.orchestra.run.smithi049.stdout:Resetting modules: 2023-07-20T17:29:36.429 INFO:teuthology.orchestra.run.smithi049.stdout: container-tools 2023-07-20T17:29:36.429 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:29:36.430 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-07-20T17:29:36.430 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-07-20T17:29:36.430 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:29:37.118 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:42 ago on Thu 20 Jul 2023 05:27:54 PM UTC. 2023-07-20T17:29:37.145 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:04:24 ago on Thu 20 Jul 2023 05:25:12 PM UTC. 2023-07-20T17:29:37.440 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:37 ago on Thu 20 Jul 2023 05:25:59 PM UTC. 2023-07-20T17:29:42.693 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-07-20T17:29:42.740 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-07-20T17:29:43.163 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-07-20T17:29:43.200 INFO:teuthology.orchestra.run.smithi079.stdout:Updating Subscription Management repositories. 2023-07-20T17:29:43.201 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-07-20T17:29:43.665 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-07-20T17:29:44.258 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:49 ago on Thu 20 Jul 2023 05:27:54 PM UTC. 2023-07-20T17:29:44.302 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:04:31 ago on Thu 20 Jul 2023 05:25:12 PM UTC. 2023-07-20T17:29:44.666 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:45 ago on Thu 20 Jul 2023 05:25:59 PM UTC. 2023-07-20T17:29:45.969 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-07-20T17:29:46.017 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-07-20T17:29:46.063 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================================ 2023-07-20T17:29:46.063 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-07-20T17:29:46.064 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================================ 2023-07-20T17:29:46.064 INFO:teuthology.orchestra.run.smithi097.stdout:Installing group/module packages: 2023-07-20T17:29:46.064 INFO:teuthology.orchestra.run.smithi097.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T17:29:46.064 INFO:teuthology.orchestra.run.smithi097.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T17:29:46.064 INFO:teuthology.orchestra.run.smithi097.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T17:29:46.064 INFO:teuthology.orchestra.run.smithi097.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T17:29:46.064 INFO:teuthology.orchestra.run.smithi097.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T17:29:46.065 INFO:teuthology.orchestra.run.smithi097.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T17:29:46.065 INFO:teuthology.orchestra.run.smithi097.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T17:29:46.065 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-07-20T17:29:46.065 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T17:29:46.065 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T17:29:46.065 INFO:teuthology.orchestra.run.smithi097.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T17:29:46.065 INFO:teuthology.orchestra.run.smithi097.stdout:Installing module profiles: 2023-07-20T17:29:46.066 INFO:teuthology.orchestra.run.smithi097.stdout: container-tools/common 2023-07-20T17:29:46.066 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:29:46.066 INFO:teuthology.orchestra.run.smithi097.stdout:Enabling module streams: 2023-07-20T17:29:46.066 INFO:teuthology.orchestra.run.smithi097.stdout: container-tools rhel8 2023-07-20T17:29:46.066 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:29:46.066 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-07-20T17:29:46.066 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================================ 2023-07-20T17:29:46.067 INFO:teuthology.orchestra.run.smithi097.stdout:Install 10 Packages 2023-07-20T17:29:46.067 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:29:46.067 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 18 M 2023-07-20T17:29:46.068 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 65 M 2023-07-20T17:29:46.068 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-07-20T17:29:46.110 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================================ 2023-07-20T17:29:46.111 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-07-20T17:29:46.111 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================================ 2023-07-20T17:29:46.111 INFO:teuthology.orchestra.run.smithi079.stdout:Installing group/module packages: 2023-07-20T17:29:46.111 INFO:teuthology.orchestra.run.smithi079.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T17:29:46.111 INFO:teuthology.orchestra.run.smithi079.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T17:29:46.111 INFO:teuthology.orchestra.run.smithi079.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T17:29:46.112 INFO:teuthology.orchestra.run.smithi079.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T17:29:46.112 INFO:teuthology.orchestra.run.smithi079.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T17:29:46.112 INFO:teuthology.orchestra.run.smithi079.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T17:29:46.112 INFO:teuthology.orchestra.run.smithi079.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T17:29:46.112 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2023-07-20T17:29:46.112 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T17:29:46.112 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T17:29:46.113 INFO:teuthology.orchestra.run.smithi079.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T17:29:46.113 INFO:teuthology.orchestra.run.smithi079.stdout:Installing module profiles: 2023-07-20T17:29:46.113 INFO:teuthology.orchestra.run.smithi079.stdout: container-tools/common 2023-07-20T17:29:46.113 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:29:46.113 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling module streams: 2023-07-20T17:29:46.113 INFO:teuthology.orchestra.run.smithi079.stdout: container-tools rhel8 2023-07-20T17:29:46.113 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:29:46.113 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-07-20T17:29:46.114 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================================ 2023-07-20T17:29:46.114 INFO:teuthology.orchestra.run.smithi079.stdout:Install 10 Packages 2023-07-20T17:29:46.114 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:29:46.114 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 18 M 2023-07-20T17:29:46.115 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 65 M 2023-07-20T17:29:46.115 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-07-20T17:29:46.140 INFO:teuthology.orchestra.run.smithi097.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 546 kB/s | 41 kB 00:00 2023-07-20T17:29:46.166 INFO:teuthology.orchestra.run.smithi097.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 248 kB/s | 25 kB 00:00 2023-07-20T17:29:46.175 INFO:teuthology.orchestra.run.smithi079.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 655 kB/s | 41 kB 00:00 2023-07-20T17:29:46.191 INFO:teuthology.orchestra.run.smithi097.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 750 kB/s | 94 kB 00:00 2023-07-20T17:29:46.192 INFO:teuthology.orchestra.run.smithi079.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 313 kB/s | 25 kB 00:00 2023-07-20T17:29:46.216 INFO:teuthology.orchestra.run.smithi097.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 656 kB/s | 49 kB 00:00 2023-07-20T17:29:46.217 INFO:teuthology.orchestra.run.smithi079.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 899 kB/s | 94 kB 00:00 2023-07-20T17:29:46.243 INFO:teuthology.orchestra.run.smithi079.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 735 kB/s | 49 kB 00:00 2023-07-20T17:29:46.249 INFO:teuthology.orchestra.run.smithi097.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.1 MB/s | 524 kB 00:00 2023-07-20T17:29:46.275 INFO:teuthology.orchestra.run.smithi097.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-07-20T17:29:46.276 INFO:teuthology.orchestra.run.smithi079.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.1 MB/s | 524 kB 00:00 2023-07-20T17:29:46.319 INFO:teuthology.orchestra.run.smithi079.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.9 MB/s | 149 kB 00:00 2023-07-20T17:29:46.333 INFO:teuthology.orchestra.run.smithi097.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 2.5 MB/s | 213 kB 00:00 2023-07-20T17:29:46.369 INFO:teuthology.orchestra.run.smithi079.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 2.3 MB/s | 213 kB 00:00 2023-07-20T17:29:46.451 INFO:teuthology.orchestra.run.smithi097.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 19 MB/s | 2.2 MB 00:00 2023-07-20T17:29:46.494 INFO:teuthology.orchestra.run.smithi079.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 29 MB/s | 8.0 MB 00:00 2023-07-20T17:29:46.552 INFO:teuthology.orchestra.run.smithi079.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 12 MB/s | 2.2 MB 00:00 2023-07-20T17:29:46.567 INFO:teuthology.orchestra.run.smithi097.stdout:(9/10): buildah-1.26.2-1.module+el8.6.0+16771+2 21 MB/s | 8.0 MB 00:00 2023-07-20T17:29:46.622 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-07-20T17:29:46.644 INFO:teuthology.orchestra.run.smithi079.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 22 MB/s | 7.0 MB 00:00 2023-07-20T17:29:46.645 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-07-20T17:29:46.645 INFO:teuthology.orchestra.run.smithi079.stdout:Total 34 MB/s | 18 MB 00:00 2023-07-20T17:29:46.684 INFO:teuthology.orchestra.run.smithi097.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 17 MB/s | 7.0 MB 00:00 2023-07-20T17:29:46.685 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-07-20T17:29:46.685 INFO:teuthology.orchestra.run.smithi097.stdout:Total 29 MB/s | 18 MB 00:00 2023-07-20T17:29:46.722 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================================ 2023-07-20T17:29:46.722 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-07-20T17:29:46.723 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================================ 2023-07-20T17:29:46.723 INFO:teuthology.orchestra.run.smithi049.stdout:Installing group/module packages: 2023-07-20T17:29:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T17:29:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T17:29:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T17:29:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T17:29:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T17:29:46.724 INFO:teuthology.orchestra.run.smithi049.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T17:29:46.724 INFO:teuthology.orchestra.run.smithi049.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T17:29:46.724 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-07-20T17:29:46.724 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T17:29:46.724 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T17:29:46.724 INFO:teuthology.orchestra.run.smithi049.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T17:29:46.724 INFO:teuthology.orchestra.run.smithi049.stdout:Installing module profiles: 2023-07-20T17:29:46.725 INFO:teuthology.orchestra.run.smithi049.stdout: container-tools/common 2023-07-20T17:29:46.725 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:29:46.725 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling module streams: 2023-07-20T17:29:46.725 INFO:teuthology.orchestra.run.smithi049.stdout: container-tools rhel8 2023-07-20T17:29:46.725 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:29:46.725 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-07-20T17:29:46.725 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================================ 2023-07-20T17:29:46.725 INFO:teuthology.orchestra.run.smithi049.stdout:Install 10 Packages 2023-07-20T17:29:46.726 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:29:46.726 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 18 M 2023-07-20T17:29:46.726 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 65 M 2023-07-20T17:29:46.727 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-07-20T17:29:46.844 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-07-20T17:29:46.855 INFO:teuthology.orchestra.run.smithi049.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 314 kB/s | 41 kB 00:00 2023-07-20T17:29:46.862 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-07-20T17:29:46.862 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-07-20T17:29:46.880 INFO:teuthology.orchestra.run.smithi049.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 160 kB/s | 25 kB 00:00 2023-07-20T17:29:46.887 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-07-20T17:29:46.905 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-07-20T17:29:46.906 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-07-20T17:29:46.906 INFO:teuthology.orchestra.run.smithi049.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 520 kB/s | 94 kB 00:00 2023-07-20T17:29:46.931 INFO:teuthology.orchestra.run.smithi049.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 653 kB/s | 49 kB 00:00 2023-07-20T17:29:46.981 INFO:teuthology.orchestra.run.smithi049.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 5.1 MB/s | 524 kB 00:00 2023-07-20T17:29:47.045 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-07-20T17:29:47.049 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-07-20T17:29:47.073 INFO:teuthology.orchestra.run.smithi049.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.0 MB/s | 149 kB 00:00 2023-07-20T17:29:47.087 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-07-20T17:29:47.091 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-07-20T17:29:47.098 INFO:teuthology.orchestra.run.smithi049.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.8 MB/s | 213 kB 00:00 2023-07-20T17:29:47.224 INFO:teuthology.orchestra.run.smithi049.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 25 MB/s | 8.0 MB 00:00 2023-07-20T17:29:47.282 INFO:teuthology.orchestra.run.smithi049.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 12 MB/s | 2.2 MB 00:00 2023-07-20T17:29:47.391 INFO:teuthology.orchestra.run.smithi049.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 22 MB/s | 7.0 MB 00:00 2023-07-20T17:29:47.392 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-07-20T17:29:47.392 INFO:teuthology.orchestra.run.smithi049.stdout:Total 27 MB/s | 18 MB 00:00 2023-07-20T17:29:47.577 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-07-20T17:29:47.597 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-07-20T17:29:47.618 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-07-20T17:29:47.618 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-07-20T17:29:47.620 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-07-20T17:29:47.769 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T17:29:47.801 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T17:29:47.813 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-07-20T17:29:47.817 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-07-20T17:29:47.968 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T17:29:48.005 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T17:29:48.163 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T17:29:48.232 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T17:29:48.339 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-07-20T17:29:48.374 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T17:29:48.481 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T17:29:48.512 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T17:29:48.701 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T17:29:48.747 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T17:29:48.850 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T17:29:48.906 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T17:29:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T17:29:49.474 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T17:29:49.498 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T17:29:49.637 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T17:29:50.253 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T17:29:50.410 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T17:29:50.510 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T17:29:50.637 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T17:29:50.710 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T17:29:50.825 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T17:29:50.904 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T17:29:51.020 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T17:29:51.106 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T17:29:51.167 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T17:29:51.347 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T17:29:51.347 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T17:29:51.348 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T17:29:51.348 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T17:29:51.348 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T17:29:51.348 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T17:29:51.348 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T17:29:51.349 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T17:29:51.349 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T17:29:51.349 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T17:29:51.366 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T17:29:51.368 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T17:29:51.368 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T17:29:51.368 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T17:29:51.368 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T17:29:51.368 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T17:29:51.368 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T17:29:51.369 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T17:29:51.369 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T17:29:51.369 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T17:29:51.396 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T17:29:51.582 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T17:29:52.008 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T17:29:52.277 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T17:29:52.277 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T17:29:52.277 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T17:29:52.277 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T17:29:52.278 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T17:29:52.278 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T17:29:52.278 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T17:29:52.278 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T17:29:52.278 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T17:29:52.278 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T17:29:52.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T17:29:52.487 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:58 ago on Thu 20 Jul 2023 05:27:54 PM UTC. 2023-07-20T17:29:52.510 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T17:29:52.511 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:04:40 ago on Thu 20 Jul 2023 05:25:12 PM UTC. 2023-07-20T17:29:53.484 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T17:29:53.484 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:54 ago on Thu 20 Jul 2023 05:25:59 PM UTC. 2023-07-20T17:29:59.516 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-07-20T17:29:59.706 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:29:59.707 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-07-20T17:29:59.707 INFO:teuthology.orchestra.run.smithi097.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:29:59.707 INFO:teuthology.orchestra.run.smithi097.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T17:29:59.707 INFO:teuthology.orchestra.run.smithi097.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:29:59.708 INFO:teuthology.orchestra.run.smithi097.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T17:29:59.708 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T17:29:59.708 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T17:29:59.708 INFO:teuthology.orchestra.run.smithi097.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:29:59.708 INFO:teuthology.orchestra.run.smithi097.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:29:59.709 INFO:teuthology.orchestra.run.smithi097.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T17:29:59.709 INFO:teuthology.orchestra.run.smithi097.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T17:29:59.709 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:29:59.709 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-07-20T17:29:59.857 DEBUG:teuthology.parallel:result is None 2023-07-20T17:30:00.829 INFO:teuthology.orchestra.run.smithi079.stdout:Installed products updated. 2023-07-20T17:30:01.024 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:30:01.024 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-07-20T17:30:01.024 INFO:teuthology.orchestra.run.smithi079.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:30:01.024 INFO:teuthology.orchestra.run.smithi079.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T17:30:01.025 INFO:teuthology.orchestra.run.smithi079.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:30:01.025 INFO:teuthology.orchestra.run.smithi079.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T17:30:01.025 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T17:30:01.025 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T17:30:01.025 INFO:teuthology.orchestra.run.smithi079.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:30:01.025 INFO:teuthology.orchestra.run.smithi079.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:30:01.025 INFO:teuthology.orchestra.run.smithi079.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi079.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-07-20T17:30:01.177 DEBUG:teuthology.parallel:result is None 2023-07-20T17:30:01.507 INFO:teuthology.orchestra.run.smithi049.stdout:Installed products updated. 2023-07-20T17:30:01.717 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:30:01.717 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-07-20T17:30:01.718 INFO:teuthology.orchestra.run.smithi049.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:30:01.718 INFO:teuthology.orchestra.run.smithi049.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T17:30:01.718 INFO:teuthology.orchestra.run.smithi049.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:30:01.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T17:30:01.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T17:30:01.719 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T17:30:01.719 INFO:teuthology.orchestra.run.smithi049.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:30:01.720 INFO:teuthology.orchestra.run.smithi049.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T17:30:01.720 INFO:teuthology.orchestra.run.smithi049.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T17:30:01.720 INFO:teuthology.orchestra.run.smithi049.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T17:30:01.720 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:30:01.720 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-07-20T17:30:01.855 DEBUG:teuthology.parallel:result is None 2023-07-20T17:30:01.856 INFO:teuthology.run_tasks:Running task install... 2023-07-20T17:30:01.866 DEBUG:teuthology.task.install:project ceph 2023-07-20T17:30:01.866 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'}} 2023-07-20T17:30:01.867 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'} 2023-07-20T17:30:01.867 INFO:teuthology.task.install:Using flavor: default 2023-07-20T17:30:01.877 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-20T17:30:01.878 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-20T17:30:01.878 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T17:30:01.879 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T17:30:01.882 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T17:30:01.882 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T17:30:01.883 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T17:30:01.884 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T17:30:02.204 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T17:30:02.205 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T17:30:02.209 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T17:30:02.210 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T17:30:02.214 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T17:30:02.215 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T17:30:02.506 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T17:30:02.507 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:30:02.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T17:30:02.511 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T17:30:02.512 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-07-20T17:30:02.512 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T17:30:02.520 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T17:30:02.520 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-07-20T17:30:02.521 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T17:30:02.539 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T17:30:02.539 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T17:30:02.549 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T17:30:02.549 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T17:30:02.554 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T17:30:02.555 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T17:30:02.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T17:30:02.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T17:30:02.626 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T17:30:02.691 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T17:30:02.703 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T17:30:02.709 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T17:30:02.759 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2023-07-20T17:30:02.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2023-07-20T17:30:02.771 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2023-07-20T17:30:02.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2023-07-20T17:30:02.777 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2023-07-20T17:30:02.778 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2023-07-20T17:30:03.146 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-07-20T17:30:03.152 INFO:teuthology.orchestra.run.smithi079.stdout:Updating Subscription Management repositories. 2023-07-20T17:30:03.153 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-07-20T17:30:04.545 INFO:teuthology.orchestra.run.smithi079.stdout:44 files removed 2023-07-20T17:30:04.561 INFO:teuthology.orchestra.run.smithi049.stdout:44 files removed 2023-07-20T17:30:04.598 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T17:30:04.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T17:30:04.763 INFO:teuthology.orchestra.run.smithi097.stdout:44 files removed 2023-07-20T17:30:04.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T17:30:04.928 INFO:teuthology.orchestra.run.smithi079.stdout:Updating Subscription Management repositories. 2023-07-20T17:30:04.968 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-07-20T17:30:05.148 INFO:teuthology.orchestra.run.smithi097.stdout:Updating Subscription Management repositories. 2023-07-20T17:30:06.987 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 121 kB/s | 81 kB 00:00 2023-07-20T17:30:07.161 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 129 kB/s | 81 kB 00:00 2023-07-20T17:30:07.252 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 94 kB/s | 81 kB 00:00 2023-07-20T17:30:07.596 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 16 kB/s | 9.4 kB 00:00 2023-07-20T17:30:07.827 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 17 kB/s | 9.4 kB 00:00 2023-07-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 3.1 kB/s | 1.5 kB 00:00 2023-07-20T17:30:08.278 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 8.7 kB/s | 9.4 kB 00:01 2023-07-20T17:30:08.336 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 3.1 kB/s | 1.5 kB 00:00 2023-07-20T17:30:08.355 INFO:teuthology.orchestra.run.smithi079.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-07-20T17:30:08.903 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 2.5 kB/s | 1.5 kB 00:00 2023-07-20T17:30:09.628 INFO:teuthology.orchestra.run.smithi097.stdout:Copr repo for python3-asyncssh owned by ceph 2.7 kB/s | 3.5 kB 00:01 2023-07-20T17:30:10.137 INFO:teuthology.orchestra.run.smithi049.stdout:Copr repo for python3-asyncssh owned by ceph 2.9 kB/s | 3.5 kB 00:01 2023-07-20T17:30:11.122 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-07-20T17:30:14.562 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 359 kB/s | 24 kB 00:00 2023-07-20T17:30:15.714 INFO:teuthology.orchestra.run.smithi097.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 MB/s | 53 MB 00:01 2023-07-20T17:30:21.053 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 1.4 MB/s | 16 MB 00:10 2023-07-20T17:30:24.074 INFO:teuthology.orchestra.run.smithi097.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 47 MB 00:01 2023-07-20T17:30:24.704 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 467 kB/s | 24 kB 00:00 2023-07-20T17:30:25.839 INFO:teuthology.orchestra.run.smithi049.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 47 MB 00:01 2023-07-20T17:30:34.648 INFO:teuthology.orchestra.run.smithi049.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 44 MB/s | 53 MB 00:01 2023-07-20T17:30:34.931 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T17:30:34.932 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T17:30:34.947 INFO:teuthology.orchestra.run.smithi097.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T17:30:34.948 INFO:teuthology.orchestra.run.smithi097.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T17:30:34.949 INFO:teuthology.orchestra.run.smithi097.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T17:30:34.950 INFO:teuthology.orchestra.run.smithi097.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T17:30:34.952 INFO:teuthology.orchestra.run.smithi097.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T17:30:34.956 INFO:teuthology.orchestra.run.smithi097.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T17:30:34.957 INFO:teuthology.orchestra.run.smithi097.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T17:30:35.087 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-07-20T17:30:35.099 INFO:teuthology.orchestra.run.smithi097.stdout:=========================================================================================================================== 2023-07-20T17:30:35.099 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-07-20T17:30:35.100 INFO:teuthology.orchestra.run.smithi097.stdout:=========================================================================================================================== 2023-07-20T17:30:35.100 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-07-20T17:30:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T17:30:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T17:30:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T17:30:35.100 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T17:30:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T17:30:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T17:30:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T17:30:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T17:30:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T17:30:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T17:30:35.101 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T17:30:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T17:30:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T17:30:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T17:30:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T17:30:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T17:30:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T17:30:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T17:30:35.102 INFO:teuthology.orchestra.run.smithi097.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T17:30:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T17:30:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T17:30:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T17:30:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T17:30:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T17:30:35.103 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T17:30:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T17:30:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T17:30:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T17:30:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T17:30:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T17:30:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T17:30:35.104 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T17:30:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T17:30:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T17:30:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T17:30:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T17:30:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T17:30:35.105 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2023-07-20T17:30:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T17:30:35.105 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T17:30:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T17:30:35.106 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-07-20T17:30:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T17:30:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T17:30:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T17:30:35.106 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T17:30:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T17:30:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T17:30:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T17:30:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T17:30:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T17:30:35.107 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T17:30:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T17:30:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T17:30:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T17:30:35.108 INFO:teuthology.orchestra.run.smithi097.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T17:30:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T17:30:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T17:30:35.109 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T17:30:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T17:30:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T17:30:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T17:30:35.110 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T17:30:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T17:30:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T17:30:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T17:30:35.111 INFO:teuthology.orchestra.run.smithi097.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T17:30:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T17:30:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T17:30:35.112 INFO:teuthology.orchestra.run.smithi097.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T17:30:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T17:30:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T17:30:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T17:30:35.113 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T17:30:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T17:30:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T17:30:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T17:30:35.114 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T17:30:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T17:30:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T17:30:35.115 INFO:teuthology.orchestra.run.smithi097.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T17:30:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T17:30:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T17:30:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T17:30:35.116 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T17:30:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T17:30:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T17:30:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T17:30:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T17:30:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T17:30:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T17:30:35.117 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T17:30:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T17:30:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T17:30:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T17:30:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T17:30:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T17:30:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T17:30:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T17:30:35.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T17:30:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T17:30:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T17:30:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T17:30:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T17:30:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T17:30:35.119 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T17:30:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T17:30:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T17:30:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T17:30:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T17:30:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T17:30:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T17:30:35.120 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T17:30:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T17:30:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T17:30:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T17:30:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T17:30:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T17:30:35.121 INFO:teuthology.orchestra.run.smithi097.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T17:30:35.122 INFO:teuthology.orchestra.run.smithi097.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T17:30:35.122 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T17:30:35.122 INFO:teuthology.orchestra.run.smithi097.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T17:30:35.122 INFO:teuthology.orchestra.run.smithi097.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T17:30:35.122 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T17:30:35.122 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T17:30:35.122 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T17:30:35.122 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T17:30:35.123 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T17:30:35.123 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T17:30:35.123 INFO:teuthology.orchestra.run.smithi097.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T17:30:35.123 INFO:teuthology.orchestra.run.smithi097.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T17:30:35.123 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T17:30:35.123 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T17:30:35.123 INFO:teuthology.orchestra.run.smithi097.stdout:Enabling module streams: 2023-07-20T17:30:35.124 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql 10 2023-07-20T17:30:35.124 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:30:35.124 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-07-20T17:30:35.124 INFO:teuthology.orchestra.run.smithi097.stdout:=========================================================================================================================== 2023-07-20T17:30:35.124 INFO:teuthology.orchestra.run.smithi097.stdout:Install 128 Packages 2023-07-20T17:30:35.124 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 3 Packages 2023-07-20T17:30:35.125 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:30:35.125 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 206 M 2023-07-20T17:30:35.125 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-07-20T17:30:35.825 INFO:teuthology.orchestra.run.smithi097.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 30 kB/s | 6.4 kB 00:00 2023-07-20T17:30:37.441 INFO:teuthology.orchestra.run.smithi097.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 526 kB/s | 850 kB 00:01 2023-07-20T17:30:37.591 INFO:teuthology.orchestra.run.smithi097.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 1.1 MB/s | 167 kB 00:00 2023-07-20T17:30:38.625 INFO:teuthology.orchestra.run.smithi097.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 2.2 MB/s | 2.3 MB 00:01 2023-07-20T17:30:38.876 INFO:teuthology.orchestra.run.smithi097.stdout:(5/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 1.6 MB/s | 5.4 MB 00:03 2023-07-20T17:30:39.084 INFO:teuthology.orchestra.run.smithi097.stdout:(6/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 3.4 MB/s | 1.6 MB 00:00 2023-07-20T17:30:39.593 INFO:teuthology.orchestra.run.smithi097.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 6.4 MB/s | 4.6 MB 00:00 2023-07-20T17:30:41.160 INFO:teuthology.orchestra.run.smithi097.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 4.0 MB/s | 22 MB 00:05 2023-07-20T17:30:41.235 INFO:teuthology.orchestra.run.smithi097.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 325 kB/s | 24 kB 00:00 2023-07-20T17:30:41.477 INFO:teuthology.orchestra.run.smithi097.stdout:(10/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 7.2 MB/s | 14 MB 00:01 2023-07-20T17:30:41.552 INFO:teuthology.orchestra.run.smithi097.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 3.4 MB/s | 260 kB 00:00 2023-07-20T17:30:41.611 INFO:teuthology.orchestra.run.smithi097.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 534 kB/s | 31 kB 00:00 2023-07-20T17:30:41.769 INFO:teuthology.orchestra.run.smithi097.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 4.8 MB/s | 771 kB 00:00 2023-07-20T17:30:41.861 INFO:teuthology.orchestra.run.smithi097.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.5 MB/s | 139 kB 00:00 2023-07-20T17:30:41.928 INFO:teuthology.orchestra.run.smithi097.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 2.0 MB/s | 139 kB 00:00 2023-07-20T17:30:42.128 INFO:teuthology.orchestra.run.smithi097.stdout:(16/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 5.8 MB/s | 18 MB 00:03 2023-07-20T17:30:42.164 INFO:teuthology.orchestra.run.smithi097.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 2.0 MB/s | 472 kB 00:00 2023-07-20T17:30:42.229 INFO:teuthology.orchestra.run.smithi097.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 672 kB/s | 45 kB 00:00 2023-07-20T17:30:42.304 INFO:teuthology.orchestra.run.smithi097.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 1.6 MB/s | 122 kB 00:00 2023-07-20T17:30:42.415 INFO:teuthology.orchestra.run.smithi097.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 1.8 MB/s | 197 kB 00:00 2023-07-20T17:30:42.505 INFO:teuthology.orchestra.run.smithi097.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 4.0 MB/s | 370 kB 00:00 2023-07-20T17:30:42.597 INFO:teuthology.orchestra.run.smithi097.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 3.9 MB/s | 363 kB 00:00 2023-07-20T17:30:42.664 INFO:teuthology.orchestra.run.smithi097.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.6 MB/s | 107 kB 00:00 2023-07-20T17:30:42.755 INFO:teuthology.orchestra.run.smithi097.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.0 MB/s | 94 kB 00:00 2023-07-20T17:30:43.031 INFO:teuthology.orchestra.run.smithi097.stdout:(25/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 7.5 MB/s | 6.8 MB 00:00 2023-07-20T17:30:43.123 INFO:teuthology.orchestra.run.smithi097.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.9 MB/s | 179 kB 00:00 2023-07-20T17:30:43.182 INFO:teuthology.orchestra.run.smithi097.stdout:(27/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 8.3 MB/s | 3.5 MB 00:00 2023-07-20T17:30:43.207 INFO:teuthology.orchestra.run.smithi097.stdout:(28/131): ceph-grafana-dashboards-18.0.0-5040.g 276 kB/s | 23 kB 00:00 2023-07-20T17:30:43.295 INFO:teuthology.orchestra.run.smithi097.stdout:(29/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.1 MB/s | 129 kB 00:00 2023-07-20T17:30:43.507 INFO:teuthology.orchestra.run.smithi097.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 5.7 MB/s | 1.7 MB 00:00 2023-07-20T17:30:43.624 INFO:teuthology.orchestra.run.smithi097.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 2.0 MB/s | 241 kB 00:00 2023-07-20T17:30:43.716 INFO:teuthology.orchestra.run.smithi097.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 518 kB/s | 47 kB 00:00 2023-07-20T17:30:43.800 INFO:teuthology.orchestra.run.smithi097.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 174 kB/s | 15 kB 00:00 2023-07-20T17:30:43.900 INFO:teuthology.orchestra.run.smithi097.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 2.5 MB/s | 258 kB 00:00 2023-07-20T17:30:43.992 INFO:teuthology.orchestra.run.smithi097.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 1.8 MB/s | 165 kB 00:00 2023-07-20T17:30:44.084 INFO:teuthology.orchestra.run.smithi097.stdout:(36/131): ceph-mgr-diskprediction-local-18.0.0- 9.4 MB/s | 7.4 MB 00:00 2023-07-20T17:30:44.109 INFO:teuthology.orchestra.run.smithi097.stdout:(37/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 154 kB/s | 18 kB 00:00 2023-07-20T17:30:44.749 INFO:teuthology.orchestra.run.smithi097.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 92 kB/s | 59 kB 00:00 2023-07-20T17:30:44.968 INFO:teuthology.orchestra.run.smithi097.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 344 kB/s | 75 kB 00:00 2023-07-20T17:30:45.002 INFO:teuthology.orchestra.run.smithi097.stdout:(40/131): gperftools-libs-2.7-9.el8.x86_64.rpm 334 kB/s | 306 kB 00:00 2023-07-20T17:30:45.152 INFO:teuthology.orchestra.run.smithi097.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 291 kB/s | 44 kB 00:00 2023-07-20T17:30:45.952 INFO:teuthology.orchestra.run.smithi097.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 484 kB/s | 476 kB 00:00 2023-07-20T17:30:46.086 INFO:teuthology.orchestra.run.smithi097.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 248 kB/s | 33 kB 00:00 2023-07-20T17:30:46.111 INFO:teuthology.orchestra.run.smithi097.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 193 kB/s | 185 kB 00:00 2023-07-20T17:30:46.228 INFO:teuthology.orchestra.run.smithi097.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 115 kB/s | 16 kB 00:00 2023-07-20T17:30:46.498 INFO:teuthology.orchestra.run.smithi097.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 447 kB/s | 173 kB 00:00 2023-07-20T17:30:46.539 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T17:30:46.540 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T17:30:46.555 INFO:teuthology.orchestra.run.smithi049.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T17:30:46.556 INFO:teuthology.orchestra.run.smithi049.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T17:30:46.557 INFO:teuthology.orchestra.run.smithi049.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T17:30:46.558 INFO:teuthology.orchestra.run.smithi049.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T17:30:46.561 INFO:teuthology.orchestra.run.smithi049.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T17:30:46.564 INFO:teuthology.orchestra.run.smithi049.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T17:30:46.565 INFO:teuthology.orchestra.run.smithi049.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T17:30:46.708 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-07-20T17:30:46.722 INFO:teuthology.orchestra.run.smithi049.stdout:=========================================================================================================================== 2023-07-20T17:30:46.722 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-07-20T17:30:46.722 INFO:teuthology.orchestra.run.smithi049.stdout:=========================================================================================================================== 2023-07-20T17:30:46.722 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-07-20T17:30:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T17:30:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T17:30:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T17:30:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T17:30:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T17:30:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T17:30:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T17:30:46.723 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T17:30:46.724 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T17:30:46.724 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T17:30:46.724 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T17:30:46.724 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T17:30:46.725 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T17:30:46.725 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T17:30:46.725 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T17:30:46.725 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T17:30:46.725 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T17:30:46.725 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T17:30:46.726 INFO:teuthology.orchestra.run.smithi049.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T17:30:46.726 INFO:teuthology.orchestra.run.smithi049.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T17:30:46.726 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T17:30:46.726 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T17:30:46.726 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T17:30:46.726 INFO:teuthology.orchestra.run.smithi049.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T17:30:46.726 INFO:teuthology.orchestra.run.smithi049.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T17:30:46.727 INFO:teuthology.orchestra.run.smithi049.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T17:30:46.727 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T17:30:46.727 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T17:30:46.727 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T17:30:46.727 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T17:30:46.727 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T17:30:46.727 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T17:30:46.727 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T17:30:46.728 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T17:30:46.728 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T17:30:46.728 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T17:30:46.728 INFO:teuthology.orchestra.run.smithi049.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T17:30:46.728 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2023-07-20T17:30:46.728 INFO:teuthology.orchestra.run.smithi049.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T17:30:46.729 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T17:30:46.729 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T17:30:46.729 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-07-20T17:30:46.729 INFO:teuthology.orchestra.run.smithi049.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T17:30:46.729 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T17:30:46.729 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T17:30:46.730 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T17:30:46.730 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T17:30:46.730 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T17:30:46.730 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T17:30:46.730 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T17:30:46.730 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T17:30:46.731 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T17:30:46.731 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T17:30:46.731 INFO:teuthology.orchestra.run.smithi049.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T17:30:46.731 INFO:teuthology.orchestra.run.smithi049.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T17:30:46.731 INFO:teuthology.orchestra.run.smithi049.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T17:30:46.731 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T17:30:46.731 INFO:teuthology.orchestra.run.smithi049.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T17:30:46.732 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T17:30:46.732 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T17:30:46.732 INFO:teuthology.orchestra.run.smithi049.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T17:30:46.732 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T17:30:46.732 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T17:30:46.733 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T17:30:46.733 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T17:30:46.733 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T17:30:46.733 INFO:teuthology.orchestra.run.smithi049.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T17:30:46.733 INFO:teuthology.orchestra.run.smithi049.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T17:30:46.733 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T17:30:46.734 INFO:teuthology.orchestra.run.smithi049.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T17:30:46.734 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T17:30:46.734 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T17:30:46.734 INFO:teuthology.orchestra.run.smithi049.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T17:30:46.734 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T17:30:46.734 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T17:30:46.734 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T17:30:46.735 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T17:30:46.735 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T17:30:46.735 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T17:30:46.735 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T17:30:46.735 INFO:teuthology.orchestra.run.smithi049.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T17:30:46.735 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T17:30:46.735 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T17:30:46.736 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T17:30:46.736 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T17:30:46.736 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T17:30:46.736 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T17:30:46.736 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T17:30:46.736 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T17:30:46.736 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T17:30:46.737 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T17:30:46.737 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T17:30:46.737 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T17:30:46.737 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T17:30:46.737 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T17:30:46.737 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T17:30:46.738 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T17:30:46.738 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T17:30:46.738 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T17:30:46.738 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T17:30:46.738 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T17:30:46.738 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T17:30:46.738 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T17:30:46.738 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T17:30:46.739 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T17:30:46.739 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T17:30:46.739 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T17:30:46.739 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T17:30:46.739 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T17:30:46.739 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T17:30:46.739 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T17:30:46.740 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T17:30:46.740 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T17:30:46.740 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T17:30:46.740 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T17:30:46.740 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T17:30:46.740 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T17:30:46.740 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T17:30:46.741 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T17:30:46.741 INFO:teuthology.orchestra.run.smithi049.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T17:30:46.741 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T17:30:46.741 INFO:teuthology.orchestra.run.smithi049.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T17:30:46.741 INFO:teuthology.orchestra.run.smithi049.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T17:30:46.741 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T17:30:46.741 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T17:30:46.742 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T17:30:46.742 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T17:30:46.742 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T17:30:46.742 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T17:30:46.742 INFO:teuthology.orchestra.run.smithi049.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T17:30:46.742 INFO:teuthology.orchestra.run.smithi049.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T17:30:46.742 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T17:30:46.742 INFO:teuthology.orchestra.run.smithi049.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T17:30:46.743 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling module streams: 2023-07-20T17:30:46.743 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql 10 2023-07-20T17:30:46.743 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:30:46.743 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-07-20T17:30:46.743 INFO:teuthology.orchestra.run.smithi049.stdout:=========================================================================================================================== 2023-07-20T17:30:46.743 INFO:teuthology.orchestra.run.smithi049.stdout:Install 128 Packages 2023-07-20T17:30:46.743 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 3 Packages 2023-07-20T17:30:46.743 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:30:46.744 INFO:teuthology.orchestra.run.smithi097.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 352 kB/s | 81 kB 00:00 2023-07-20T17:30:46.745 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 206 M 2023-07-20T17:30:46.745 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-07-20T17:30:46.846 INFO:teuthology.orchestra.run.smithi097.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2023-07-20T17:30:46.963 INFO:teuthology.orchestra.run.smithi097.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 151 kB/s | 18 kB 00:00 2023-07-20T17:30:46.988 INFO:teuthology.orchestra.run.smithi097.stdout:(50/131): python3-cherrypy-18.4.0-1.el8.noarch. 506 kB/s | 384 kB 00:00 2023-07-20T17:30:47.065 INFO:teuthology.orchestra.run.smithi049.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 28 kB/s | 6.4 kB 00:00 2023-07-20T17:30:47.105 INFO:teuthology.orchestra.run.smithi097.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 418 kB/s | 49 kB 00:00 2023-07-20T17:30:47.222 INFO:teuthology.orchestra.run.smithi097.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 504 kB/s | 59 kB 00:00 2023-07-20T17:30:47.366 INFO:teuthology.orchestra.run.smithi097.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 417 kB/s | 60 kB 00:00 2023-07-20T17:30:47.540 INFO:teuthology.orchestra.run.smithi049.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.7 MB/s | 850 kB 00:00 2023-07-20T17:30:47.691 INFO:teuthology.orchestra.run.smithi049.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 1.1 MB/s | 167 kB 00:00 2023-07-20T17:30:48.256 INFO:teuthology.orchestra.run.smithi097.stdout:(54/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8.x 6.8 MB/s | 48 MB 00:07 2023-07-20T17:30:48.314 INFO:teuthology.orchestra.run.smithi097.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 298 kB/s | 283 kB 00:00 2023-07-20T17:30:48.332 INFO:teuthology.orchestra.run.smithi049.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 3.6 MB/s | 2.3 MB 00:00 2023-07-20T17:30:48.456 INFO:teuthology.orchestra.run.smithi097.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 237 kB/s | 34 kB 00:00 2023-07-20T17:30:48.457 INFO:teuthology.orchestra.run.smithi049.stdout:(5/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 3.3 MB/s | 5.4 MB 00:01 2023-07-20T17:30:48.791 INFO:teuthology.orchestra.run.smithi049.stdout:(6/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 3.4 MB/s | 1.6 MB 00:00 2023-07-20T17:30:48.808 INFO:teuthology.orchestra.run.smithi097.stdout:(57/131): python3-portend-2.6-1.el8.noarch.rpm 28 kB/s | 16 kB 00:00 2023-07-20T17:30:48.908 INFO:teuthology.orchestra.run.smithi097.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 435 kB/s | 196 kB 00:00 2023-07-20T17:30:49.052 INFO:teuthology.orchestra.run.smithi097.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 132 kB/s | 19 kB 00:00 2023-07-20T17:30:49.076 INFO:teuthology.orchestra.run.smithi097.stdout:(60/131): python3-rsa-4.8-1.el8.noarch.rpm 230 kB/s | 61 kB 00:00 2023-07-20T17:30:49.091 INFO:teuthology.orchestra.run.smithi049.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 7.3 MB/s | 4.6 MB 00:00 2023-07-20T17:30:49.109 INFO:teuthology.orchestra.run.smithi097.stdout:(61/131): python3-kubernetes-11.0.0-6.el8.noarc 556 kB/s | 1.2 MB 00:02 2023-07-20T17:30:49.195 INFO:teuthology.orchestra.run.smithi097.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 168 kB/s | 24 kB 00:00 2023-07-20T17:30:49.210 INFO:teuthology.orchestra.run.smithi097.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 218 kB/s | 29 kB 00:00 2023-07-20T17:30:49.227 INFO:teuthology.orchestra.run.smithi097.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 235 kB/s | 27 kB 00:00 2023-07-20T17:30:49.360 INFO:teuthology.orchestra.run.smithi097.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 459 kB/s | 61 kB 00:00 2023-07-20T17:30:49.636 INFO:teuthology.orchestra.run.smithi097.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 310 kB/s | 85 kB 00:00 2023-07-20T17:30:49.661 INFO:teuthology.orchestra.run.smithi097.stdout:(67/131): python3-waitress-1.4.3-1.el8.noarch.r 546 kB/s | 255 kB 00:00 2023-07-20T17:30:49.786 INFO:teuthology.orchestra.run.smithi097.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 153 kB/s | 23 kB 00:00 2023-07-20T17:30:49.811 INFO:teuthology.orchestra.run.smithi097.stdout:(69/131): python3-webob-1.8.5-1.el8.1.noarch.rp 418 kB/s | 251 kB 00:00 2023-07-20T17:30:49.845 INFO:teuthology.orchestra.run.smithi097.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 70 kB 00:00 2023-07-20T17:30:50.035 INFO:teuthology.orchestra.run.smithi097.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 210 kB/s | 47 kB 00:00 2023-07-20T17:30:50.071 INFO:teuthology.orchestra.run.smithi097.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 910 kB/s | 200 kB 00:00 2023-07-20T17:30:50.180 INFO:teuthology.orchestra.run.smithi097.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 294 kB/s | 43 kB 00:00 2023-07-20T17:30:50.205 INFO:teuthology.orchestra.run.smithi097.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2023-07-20T17:30:50.339 INFO:teuthology.orchestra.run.smithi097.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 302 kB/s | 48 kB 00:00 2023-07-20T17:30:50.372 INFO:teuthology.orchestra.run.smithi097.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 522 kB/s | 87 kB 00:00 2023-07-20T17:30:50.447 INFO:teuthology.orchestra.run.smithi097.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 397 kB/s | 43 kB 00:00 2023-07-20T17:30:50.458 INFO:teuthology.orchestra.run.smithi049.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 6.1 MB/s | 22 MB 00:03 2023-07-20T17:30:50.508 INFO:teuthology.orchestra.run.smithi049.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 487 kB/s | 24 kB 00:00 2023-07-20T17:30:50.539 INFO:teuthology.orchestra.run.smithi097.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 653 kB/s | 109 kB 00:00 2023-07-20T17:30:50.640 INFO:teuthology.orchestra.run.smithi097.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.2 MB/s | 229 kB 00:00 2023-07-20T17:30:50.650 INFO:teuthology.orchestra.run.smithi049.stdout:(10/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 8.7 MB/s | 14 MB 00:01 2023-07-20T17:30:50.673 INFO:teuthology.orchestra.run.smithi097.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.1 MB/s | 560 kB 00:00 2023-07-20T17:30:50.734 INFO:teuthology.orchestra.run.smithi049.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 3.1 MB/s | 260 kB 00:00 2023-07-20T17:30:50.757 INFO:teuthology.orchestra.run.smithi097.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 160 kB/s | 19 kB 00:00 2023-07-20T17:30:50.801 INFO:teuthology.orchestra.run.smithi049.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 467 kB/s | 31 kB 00:00 2023-07-20T17:30:50.807 INFO:teuthology.orchestra.run.smithi097.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.1 MB/s | 155 kB 00:00 2023-07-20T17:30:50.865 INFO:teuthology.orchestra.run.smithi097.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 5.0 MB/s | 559 kB 00:00 2023-07-20T17:30:50.909 INFO:teuthology.orchestra.run.smithi049.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 7.0 MB/s | 771 kB 00:00 2023-07-20T17:30:50.949 INFO:teuthology.orchestra.run.smithi097.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 126 kB/s | 18 kB 00:00 2023-07-20T17:30:51.009 INFO:teuthology.orchestra.run.smithi049.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.4 MB/s | 139 kB 00:00 2023-07-20T17:30:51.033 INFO:teuthology.orchestra.run.smithi097.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.5 MB/s | 605 kB 00:00 2023-07-20T17:30:51.058 INFO:teuthology.orchestra.run.smithi097.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.1 MB/s | 238 kB 00:00 2023-07-20T17:30:51.085 INFO:teuthology.orchestra.run.smithi049.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.8 MB/s | 139 kB 00:00 2023-07-20T17:30:51.150 INFO:teuthology.orchestra.run.smithi097.stdout:(87/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 20 MB/s | 2.3 MB 00:00 2023-07-20T17:30:51.177 INFO:teuthology.orchestra.run.smithi049.stdout:(16/131): libradosstriper1-18.0.0-5040.g1b0b2eb 5.0 MB/s | 472 kB 00:00 2023-07-20T17:30:51.234 INFO:teuthology.orchestra.run.smithi097.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 413 kB/s | 72 kB 00:00 2023-07-20T17:30:51.334 INFO:teuthology.orchestra.run.smithi097.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 451 kB/s | 83 kB 00:00 2023-07-20T17:30:51.351 INFO:teuthology.orchestra.run.smithi097.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 512 kB/s | 60 kB 00:00 2023-07-20T17:30:51.426 INFO:teuthology.orchestra.run.smithi097.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.0 MB/s | 98 kB 00:00 2023-07-20T17:30:51.518 INFO:teuthology.orchestra.run.smithi097.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 345 kB/s | 58 kB 00:00 2023-07-20T17:30:51.544 INFO:teuthology.orchestra.run.smithi097.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 330 kB/s | 39 kB 00:00 2023-07-20T17:30:51.694 INFO:teuthology.orchestra.run.smithi097.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.1 MB/s | 171 kB 00:00 2023-07-20T17:30:51.727 INFO:teuthology.orchestra.run.smithi097.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 3.0 MB/s | 644 kB 00:00 2023-07-20T17:30:51.794 INFO:teuthology.orchestra.run.smithi049.stdout:(17/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 5.9 MB/s | 18 MB 00:03 2023-07-20T17:30:51.798 INFO:teuthology.orchestra.run.smithi097.stdout:(96/131): NetworkManager-initscripts-updown-1.3 1.3 MB/s | 138 kB 00:00 2023-07-20T17:30:51.911 INFO:teuthology.orchestra.run.smithi049.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 383 kB/s | 45 kB 00:00 2023-07-20T17:30:51.936 INFO:teuthology.orchestra.run.smithi097.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 801 kB/s | 110 kB 00:00 2023-07-20T17:30:52.044 INFO:teuthology.orchestra.run.smithi049.stdout:(19/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 7.8 MB/s | 6.8 MB 00:00 2023-07-20T17:30:52.061 INFO:teuthology.orchestra.run.smithi049.stdout:(20/131): python3-ceph-common-18.0.0-5040.g1b0b 816 kB/s | 122 kB 00:00 2023-07-20T17:30:52.120 INFO:teuthology.orchestra.run.smithi097.stdout:(98/131): python3-scipy-1.0.0-21.module+el8.5.0 35 MB/s | 14 MB 00:00 2023-07-20T17:30:52.136 INFO:teuthology.orchestra.run.smithi049.stdout:(21/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 2.1 MB/s | 197 kB 00:00 2023-07-20T17:30:52.145 INFO:teuthology.orchestra.run.smithi097.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 260 kB/s | 54 kB 00:00 2023-07-20T17:30:52.170 INFO:teuthology.orchestra.run.smithi097.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.0 MB/s | 259 kB 00:00 2023-07-20T17:30:52.196 INFO:teuthology.orchestra.run.smithi097.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-07-20T17:30:52.220 INFO:teuthology.orchestra.run.smithi049.stdout:(22/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 2.3 MB/s | 370 kB 00:00 2023-07-20T17:30:52.221 INFO:teuthology.orchestra.run.smithi097.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 774 kB/s | 39 kB 00:00 2023-07-20T17:30:52.245 INFO:teuthology.orchestra.run.smithi049.stdout:(23/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 3.3 MB/s | 363 kB 00:00 2023-07-20T17:30:52.255 INFO:teuthology.orchestra.run.smithi097.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 11 MB/s | 688 kB 00:00 2023-07-20T17:30:52.288 INFO:teuthology.orchestra.run.smithi097.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 7.8 MB/s | 538 kB 00:00 2023-07-20T17:30:52.305 INFO:teuthology.orchestra.run.smithi049.stdout:(24/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.2 MB/s | 107 kB 00:00 2023-07-20T17:30:52.313 INFO:teuthology.orchestra.run.smithi097.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-07-20T17:30:52.329 INFO:teuthology.orchestra.run.smithi049.stdout:(25/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.1 MB/s | 94 kB 00:00 2023-07-20T17:30:52.347 INFO:teuthology.orchestra.run.smithi097.stdout:(106/131): thrift-0.13.0-2.el8.x86_64.rpm 661 kB/s | 1.7 MB 00:02 2023-07-20T17:30:52.406 INFO:teuthology.orchestra.run.smithi097.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-07-20T17:30:52.429 INFO:teuthology.orchestra.run.smithi049.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.7 MB/s | 179 kB 00:00 2023-07-20T17:30:52.448 INFO:teuthology.orchestra.run.smithi097.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 5.2 MB/s | 713 kB 00:00 2023-07-20T17:30:52.473 INFO:teuthology.orchestra.run.smithi097.stdout:(109/131): python3-jsonpatch-1.21-2.el8.noarch. 432 kB/s | 27 kB 00:00 2023-07-20T17:30:52.507 INFO:teuthology.orchestra.run.smithi097.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 5.3 MB/s | 320 kB 00:00 2023-07-20T17:30:52.521 INFO:teuthology.orchestra.run.smithi049.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 251 kB/s | 23 kB 00:00 2023-07-20T17:30:52.532 INFO:teuthology.orchestra.run.smithi097.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 751 kB/s | 44 kB 00:00 2023-07-20T17:30:52.582 INFO:teuthology.orchestra.run.smithi097.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 8.9 MB/s | 457 kB 00:00 2023-07-20T17:30:52.684 INFO:teuthology.orchestra.run.smithi097.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 21 MB/s | 3.7 MB 00:00 2023-07-20T17:30:52.716 INFO:teuthology.orchestra.run.smithi097.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.6 MB/s | 225 kB 00:00 2023-07-20T17:30:52.800 INFO:teuthology.orchestra.run.smithi097.stdout:(115/131): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2023-07-20T17:30:52.825 INFO:teuthology.orchestra.run.smithi097.stdout:(116/131): uuid-1.6.2-43.el8.x86_64.rpm 456 kB/s | 64 kB 00:00 2023-07-20T17:30:52.850 INFO:teuthology.orchestra.run.smithi097.stdout:(117/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 943 kB/s | 126 kB 00:00 2023-07-20T17:30:52.897 INFO:teuthology.orchestra.run.smithi049.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 345 kB/s | 129 kB 00:00 2023-07-20T17:30:52.933 INFO:teuthology.orchestra.run.smithi097.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 952 kB/s | 103 kB 00:00 2023-07-20T17:30:52.981 INFO:teuthology.orchestra.run.smithi097.stdout:(119/131): python3-jsonpointer-1.10-11.el8.noar 110 kB/s | 20 kB 00:00 2023-07-20T17:30:53.018 INFO:teuthology.orchestra.run.smithi097.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 4.1 MB/s | 709 kB 00:00 2023-07-20T17:30:53.043 INFO:teuthology.orchestra.run.smithi097.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 3.1 MB/s | 198 kB 00:00 2023-07-20T17:30:53.101 INFO:teuthology.orchestra.run.smithi097.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 3.4 MB/s | 202 kB 00:00 2023-07-20T17:30:53.122 INFO:teuthology.orchestra.run.smithi049.stdout:(29/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 4.3 MB/s | 3.5 MB 00:00 2023-07-20T17:30:53.185 INFO:teuthology.orchestra.run.smithi097.stdout:(123/131): openblas-threads-0.3.15-3.el8.x86_64 19 MB/s | 4.9 MB 00:00 2023-07-20T17:30:53.214 INFO:teuthology.orchestra.run.smithi049.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 5.4 MB/s | 1.7 MB 00:00 2023-07-20T17:30:53.276 INFO:teuthology.orchestra.run.smithi097.stdout:(124/131): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-07-20T17:30:53.298 INFO:teuthology.orchestra.run.smithi049.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 2.8 MB/s | 241 kB 00:00 2023-07-20T17:30:53.310 INFO:teuthology.orchestra.run.smithi097.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.7 MB/s | 354 kB 00:00 2023-07-20T17:30:53.356 INFO:teuthology.orchestra.run.smithi049.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 814 kB/s | 47 kB 00:00 2023-07-20T17:30:53.410 INFO:teuthology.orchestra.run.smithi097.stdout:(126/131): postgresql-contrib-10.21-2.module+el 5.9 MB/s | 808 kB 00:00 2023-07-20T17:30:53.424 INFO:teuthology.orchestra.run.smithi049.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 218 kB/s | 15 kB 00:00 2023-07-20T17:30:53.477 INFO:teuthology.orchestra.run.smithi097.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 9.0 MB/s | 1.5 MB 00:00 2023-07-20T17:30:53.524 INFO:teuthology.orchestra.run.smithi049.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 2.5 MB/s | 258 kB 00:00 2023-07-20T17:30:53.591 INFO:teuthology.orchestra.run.smithi049.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 2.4 MB/s | 165 kB 00:00 2023-07-20T17:30:53.603 INFO:teuthology.orchestra.run.smithi097.stdout:(128/131): postgresql-server-10.21-2.module+el8 12 MB/s | 5.1 MB 00:00 2023-07-20T17:30:53.649 INFO:teuthology.orchestra.run.smithi049.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 309 kB/s | 18 kB 00:00 2023-07-20T17:30:53.883 INFO:teuthology.orchestra.run.smithi097.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 2.8 MB/s | 798 kB 00:00 2023-07-20T17:30:54.129 INFO:teuthology.orchestra.run.smithi097.stdout:(130/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 5.6 MB/s | 3.6 MB 00:00 2023-07-20T17:30:54.221 INFO:teuthology.orchestra.run.smithi097.stdout:(131/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 4.5 MB/s | 3.7 MB 00:00 2023-07-20T17:30:54.225 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-07-20T17:30:54.225 INFO:teuthology.orchestra.run.smithi097.stdout:Total 11 MB/s | 206 MB 00:19 2023-07-20T17:30:54.708 INFO:teuthology.orchestra.run.smithi049.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 4.7 MB/s | 7.4 MB 00:01 2023-07-20T17:30:55.039 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-07-20T17:30:55.159 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-07-20T17:30:55.159 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-07-20T17:30:56.283 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-07-20T17:30:56.287 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-07-20T17:31:01.124 INFO:teuthology.orchestra.run.smithi049.stdout:(38/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8.x 4.5 MB/s | 48 MB 00:10 2023-07-20T17:31:01.458 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-07-20T17:31:01.664 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T17:31:01.897 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T17:31:02.091 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T17:31:02.199 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T17:31:02.441 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T17:31:02.847 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T17:31:03.074 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T17:31:03.481 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T17:31:03.666 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T17:31:03.795 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T17:31:03.982 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T17:31:04.075 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T17:31:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T17:31:04.693 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T17:31:04.967 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T17:31:05.287 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T17:31:05.701 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T17:31:05.927 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T17:31:06.074 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T17:31:06.165 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T17:31:06.332 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T17:31:06.473 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T17:31:06.719 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T17:31:06.921 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T17:31:07.088 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T17:31:07.248 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T17:31:07.369 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T17:31:07.548 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T17:31:08.163 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T17:31:08.357 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T17:31:08.936 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T17:31:09.123 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T17:31:09.814 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T17:31:10.438 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T17:31:10.625 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T17:31:10.879 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T17:31:11.080 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T17:31:11.202 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T17:31:11.442 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T17:31:11.681 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T17:31:13.416 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T17:31:14.446 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T17:31:14.794 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T17:31:15.057 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T17:31:15.292 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T17:31:15.525 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T17:31:15.716 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T17:31:15.861 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T17:31:16.029 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T17:31:16.154 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T17:31:16.323 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T17:31:16.480 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T17:31:16.582 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T17:31:16.800 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T17:31:17.025 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T17:31:17.197 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T17:31:17.260 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T17:31:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T17:31:17.659 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T17:31:17.862 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T17:31:18.067 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T17:31:18.233 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T17:31:18.409 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T17:31:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T17:31:18.812 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T17:31:19.203 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T17:31:19.523 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T17:31:19.741 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T17:31:19.975 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T17:31:20.243 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T17:31:20.581 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T17:31:20.821 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T17:31:21.023 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T17:31:21.276 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T17:31:21.484 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T17:31:21.683 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T17:31:21.989 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T17:31:22.243 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T17:31:22.459 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T17:31:22.723 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T17:31:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T17:31:23.058 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T17:31:23.350 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T17:31:23.595 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T17:31:23.759 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T17:31:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T17:31:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T17:31:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T17:31:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T17:31:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T17:31:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T17:31:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T17:31:25.301 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T17:31:25.502 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T17:31:25.719 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T17:31:25.987 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T17:31:26.161 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T17:31:26.414 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T17:31:27.541 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T17:31:27.791 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T17:31:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T17:31:28.388 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T17:31:28.667 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T17:31:28.879 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T17:31:29.647 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T17:31:29.787 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T17:31:29.890 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T17:31:30.037 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T17:31:33.751 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T17:31:33.946 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T17:31:34.537 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T17:31:34.889 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T17:31:34.959 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T17:31:35.128 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T17:31:53.833 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T17:31:53.833 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2023-07-20T17:31:53.833 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2023-07-20T17:31:53.833 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /dev 2023-07-20T17:31:53.833 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /run 2023-07-20T17:31:53.833 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2023-07-20T17:31:53.834 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2023-07-20T17:31:53.834 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2023-07-20T17:31:53.834 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2023-07-20T17:31:53.834 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /dev 2023-07-20T17:31:53.834 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:31:54.184 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T17:31:55.769 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T17:31:55.919 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T17:31:56.137 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T17:31:56.342 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T17:31:56.795 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T17:31:56.975 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T17:31:56.992 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T17:31:58.734 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T17:31:58.963 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T17:31:59.089 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T17:31:59.314 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T17:31:59.533 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T17:31:59.746 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T17:31:59.968 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T17:32:00.925 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T17:32:01.107 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T17:32:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T17:32:01.596 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T17:32:01.801 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T17:32:01.835 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T17:32:01.999 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T17:32:02.011 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T17:32:03.303 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T17:32:03.510 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T17:32:03.543 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T17:32:03.716 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T17:32:03.758 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T17:32:03.891 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T17:32:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T17:32:04.375 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T17:32:08.918 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T17:32:09.812 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T17:32:10.155 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T17:32:10.318 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T17:32:10.601 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T17:32:10.773 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T17:32:10.975 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T17:32:11.211 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T17:32:11.419 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T17:32:11.599 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T17:32:13.901 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T17:32:14.156 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T17:32:14.176 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T17:32:14.449 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T17:32:14.664 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T17:32:14.996 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T17:32:15.362 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T17:32:15.572 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T17:32:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T17:32:15.864 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T17:32:15.877 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T17:32:16.106 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T17:32:16.115 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T17:32:16.233 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T17:32:16.233 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T17:32:16.459 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T17:32:16.460 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T17:32:16.712 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T17:32:16.938 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T17:32:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T17:32:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T17:32:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T17:32:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T17:32:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T17:32:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T17:32:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T17:32:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T17:32:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T17:32:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T17:32:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T17:32:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T17:32:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T17:32:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T17:32:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T17:32:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T17:32:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T17:32:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T17:32:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T17:32:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T17:32:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T17:32:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T17:32:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T17:32:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T17:32:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T17:32:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T17:32:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T17:32:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T17:32:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T17:32:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T17:32:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T17:32:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T17:32:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T17:32:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T17:32:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T17:32:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T17:32:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T17:32:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T17:32:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T17:32:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T17:32:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T17:32:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T17:32:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T17:32:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T17:32:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T17:32:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T17:32:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T17:32:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T17:32:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T17:32:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T17:32:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T17:32:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T17:32:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T17:32:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T17:32:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T17:32:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T17:32:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T17:32:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T17:32:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T17:32:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T17:32:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T17:32:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T17:32:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T17:32:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T17:32:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T17:32:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T17:32:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T17:32:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T17:32:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T17:32:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T17:32:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T17:32:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T17:32:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T17:32:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T17:32:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T17:32:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T17:32:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T17:32:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T17:32:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T17:32:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T17:32:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T17:32:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T17:32:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T17:32:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T17:32:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T17:32:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T17:32:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T17:32:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T17:32:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T17:32:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T17:32:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T17:32:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T17:32:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T17:32:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T17:32:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T17:32:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T17:32:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T17:32:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T17:32:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T17:32:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T17:32:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T17:32:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T17:32:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T17:32:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T17:32:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T17:32:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T17:32:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T17:32:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T17:32:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T17:32:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T17:32:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T17:32:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T17:32:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T17:32:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T17:32:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T17:32:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T17:32:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T17:32:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T17:32:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T17:32:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T17:32:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T17:32:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T17:32:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T17:32:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T17:32:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T17:32:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T17:32:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T17:32:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T17:32:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T17:32:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T17:32:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T17:32:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T17:32:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T17:32:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T17:32:20.136 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T17:32:20.136 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:57 ago on Thu 20 Jul 2023 05:30:23 PM UTC. 2023-07-20T17:32:25.975 INFO:teuthology.orchestra.run.smithi097.stdout:Installed products updated. 2023-07-20T17:32:26.189 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:32:26.189 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2023-07-20T17:32:26.190 INFO:teuthology.orchestra.run.smithi097.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T17:32:26.190 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.190 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.190 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-07-20T17:32:26.190 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T17:32:26.190 INFO:teuthology.orchestra.run.smithi097.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T17:32:26.191 INFO:teuthology.orchestra.run.smithi097.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T17:32:26.191 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.191 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.191 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.191 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.191 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:32:26.191 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.191 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.192 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.192 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:32:26.192 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:32:26.192 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:32:26.193 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:32:26.193 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:32:26.193 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.193 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.194 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:32:26.194 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.194 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.194 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.194 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:32:26.194 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:32:26.194 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.195 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:32:26.195 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T17:32:26.195 INFO:teuthology.orchestra.run.smithi097.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T17:32:26.195 INFO:teuthology.orchestra.run.smithi097.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T17:32:26.195 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T17:32:26.195 INFO:teuthology.orchestra.run.smithi097.stdout: jq-1.6-3.el8.x86_64 2023-07-20T17:32:26.195 INFO:teuthology.orchestra.run.smithi097.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T17:32:26.195 INFO:teuthology.orchestra.run.smithi097.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T17:32:26.196 INFO:teuthology.orchestra.run.smithi097.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T17:32:26.196 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.196 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.196 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.196 INFO:teuthology.orchestra.run.smithi097.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T17:32:26.196 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T17:32:26.197 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T17:32:26.197 INFO:teuthology.orchestra.run.smithi097.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T17:32:26.197 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T17:32:26.197 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T17:32:26.197 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.197 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.197 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T17:32:26.198 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.198 INFO:teuthology.orchestra.run.smithi097.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T17:32:26.198 INFO:teuthology.orchestra.run.smithi097.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T17:32:26.198 INFO:teuthology.orchestra.run.smithi097.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T17:32:26.198 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T17:32:26.199 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T17:32:26.199 INFO:teuthology.orchestra.run.smithi097.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T17:32:26.199 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T17:32:26.199 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T17:32:26.199 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T17:32:26.199 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T17:32:26.199 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T17:32:26.200 INFO:teuthology.orchestra.run.smithi097.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T17:32:26.200 INFO:teuthology.orchestra.run.smithi097.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T17:32:26.200 INFO:teuthology.orchestra.run.smithi097.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T17:32:26.200 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T17:32:26.201 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T17:32:26.201 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T17:32:26.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T17:32:26.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T17:32:26.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T17:32:26.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T17:32:26.202 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T17:32:26.202 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.202 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.202 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.202 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T17:32:26.202 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T17:32:26.203 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T17:32:26.203 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T17:32:26.203 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T17:32:26.204 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T17:32:26.204 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T17:32:26.204 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T17:32:26.204 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T17:32:26.204 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T17:32:26.205 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T17:32:26.205 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T17:32:26.205 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T17:32:26.205 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T17:32:26.205 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T17:32:26.205 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T17:32:26.205 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T17:32:26.205 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T17:32:26.206 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T17:32:26.206 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T17:32:26.206 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T17:32:26.206 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T17:32:26.206 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T17:32:26.206 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T17:32:26.206 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T17:32:26.207 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T17:32:26.207 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T17:32:26.207 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T17:32:26.207 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T17:32:26.207 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.207 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.208 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T17:32:26.208 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T17:32:26.208 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.208 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T17:32:26.208 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T17:32:26.208 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T17:32:26.208 INFO:teuthology.orchestra.run.smithi097.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T17:32:26.209 INFO:teuthology.orchestra.run.smithi097.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T17:32:26.209 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T17:32:26.209 INFO:teuthology.orchestra.run.smithi097.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T17:32:26.209 INFO:teuthology.orchestra.run.smithi097.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T17:32:26.209 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T17:32:26.209 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T17:32:26.210 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T17:32:26.210 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T17:32:26.210 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T17:32:26.210 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.210 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.210 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:32:26.210 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T17:32:26.211 INFO:teuthology.orchestra.run.smithi097.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T17:32:26.211 INFO:teuthology.orchestra.run.smithi097.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T17:32:26.211 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T17:32:26.211 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T17:32:26.212 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T17:32:26.212 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:32:26.212 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-07-20T17:32:26.381 DEBUG:teuthology.parallel:result is None 2023-07-20T17:33:02.688 INFO:teuthology.orchestra.run.smithi049.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-07-20T17:33:03.233 INFO:teuthology.orchestra.run.smithi049.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 2.4 kB/s | 306 kB 02:09 2023-07-20T17:33:06.784 INFO:teuthology.orchestra.run.smithi049.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-07-20T17:33:06.999 INFO:teuthology.orchestra.run.smithi049.stdout:(40/131): liboath-2.6.2-3.el8.x86_64.rpm 455 B/s | 59 kB 02:12 2023-07-20T17:33:11.203 INFO:teuthology.orchestra.run.smithi049.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-07-20T17:33:11.515 INFO:teuthology.orchestra.run.smithi049.stdout:(41/131): libunwind-1.3.1-3.el8.x86_64.rpm 589 B/s | 75 kB 02:10 2023-07-20T17:33:11.833 INFO:teuthology.orchestra.run.smithi049.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 584 kB/s | 185 kB 00:00 2023-07-20T17:33:11.924 INFO:teuthology.orchestra.run.smithi049.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 359 kB/s | 33 kB 00:00 2023-07-20T17:33:12.025 INFO:teuthology.orchestra.run.smithi049.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 163 kB/s | 16 kB 00:00 2023-07-20T17:33:12.391 INFO:teuthology.orchestra.run.smithi049.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 471 kB/s | 173 kB 00:00 2023-07-20T17:33:13.067 INFO:teuthology.orchestra.run.smithi049.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 570 kB/s | 384 kB 00:00 2023-07-20T17:33:13.275 INFO:teuthology.orchestra.run.smithi049.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 390 kB/s | 81 kB 00:00 2023-07-20T17:33:13.367 INFO:teuthology.orchestra.run.smithi049.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2023-07-20T17:33:13.459 INFO:teuthology.orchestra.run.smithi049.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 192 kB/s | 18 kB 00:00 2023-07-20T17:33:15.768 INFO:teuthology.orchestra.run.smithi049.stdout:(50/131): python3-kubernetes-11.0.0-6.el8.noarc 517 kB/s | 1.2 MB 00:02 2023-07-20T17:33:15.926 INFO:teuthology.orchestra.run.smithi049.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 308 kB/s | 49 kB 00:00 2023-07-20T17:33:16.244 INFO:teuthology.orchestra.run.smithi049.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 185 kB/s | 59 kB 00:00 2023-07-20T17:33:16.476 INFO:teuthology.orchestra.run.smithi049.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 258 kB/s | 60 kB 00:00 2023-07-20T17:33:17.360 INFO:teuthology.orchestra.run.smithi049.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 320 kB/s | 283 kB 00:00 2023-07-20T17:33:17.435 INFO:teuthology.orchestra.run.smithi049.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 208 kB/s | 16 kB 00:00 2023-07-20T17:33:17.585 INFO:teuthology.orchestra.run.smithi049.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 224 kB/s | 34 kB 00:00 2023-07-20T17:33:18.061 INFO:teuthology.orchestra.run.smithi049.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 414 kB/s | 196 kB 00:00 2023-07-20T17:33:18.278 INFO:teuthology.orchestra.run.smithi049.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 284 kB/s | 61 kB 00:00 2023-07-20T17:33:18.369 INFO:teuthology.orchestra.run.smithi049.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 204 kB/s | 19 kB 00:00 2023-07-20T17:33:18.486 INFO:teuthology.orchestra.run.smithi049.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 205 kB/s | 24 kB 00:00 2023-07-20T17:33:18.695 INFO:teuthology.orchestra.run.smithi049.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 140 kB/s | 29 kB 00:00 2023-07-20T17:33:18.854 INFO:teuthology.orchestra.run.smithi049.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 174 kB/s | 27 kB 00:00 2023-07-20T17:33:19.570 INFO:teuthology.orchestra.run.smithi049.stdout:(63/131): python3-waitress-1.4.3-1.el8.noarch.r 356 kB/s | 255 kB 00:00 2023-07-20T17:33:20.046 INFO:teuthology.orchestra.run.smithi049.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 529 kB/s | 251 kB 00:00 2023-07-20T17:33:20.212 INFO:teuthology.orchestra.run.smithi049.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 368 kB/s | 61 kB 00:00 2023-07-20T17:33:20.454 INFO:teuthology.orchestra.run.smithi049.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 353 kB/s | 85 kB 00:00 2023-07-20T17:33:20.596 INFO:teuthology.orchestra.run.smithi049.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 162 kB/s | 23 kB 00:00 2023-07-20T17:33:25.571 INFO:teuthology.orchestra.run.smithi049.stdout:(68/131): thrift-0.13.0-2.el8.x86_64.rpm 357 kB/s | 1.7 MB 00:04 2023-07-20T17:33:25.596 INFO:teuthology.orchestra.run.smithi049.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T17:33:26.038 INFO:teuthology.orchestra.run.smithi049.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 31 MB/s | 14 MB 00:00 2023-07-20T17:33:26.130 INFO:teuthology.orchestra.run.smithi049.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.2 MB/s | 110 kB 00:00 2023-07-20T17:33:26.247 INFO:teuthology.orchestra.run.smithi049.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 464 kB/s | 54 kB 00:00 2023-07-20T17:33:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.7 MB/s | 259 kB 00:00 2023-07-20T17:33:26.531 INFO:teuthology.orchestra.run.smithi049.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2023-07-20T17:33:26.656 INFO:teuthology.orchestra.run.smithi049.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 311 kB/s | 39 kB 00:00 2023-07-20T17:33:26.773 INFO:teuthology.orchestra.run.smithi049.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 5.8 MB/s | 688 kB 00:00 2023-07-20T17:33:26.940 INFO:teuthology.orchestra.run.smithi049.stdout:(77/131): python3-jinja2-2.10.1-3.el8.noarch.rp 3.2 MB/s | 538 kB 00:00 2023-07-20T17:33:27.049 INFO:teuthology.orchestra.run.smithi049.stdout:(78/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2023-07-20T17:33:27.208 INFO:teuthology.orchestra.run.smithi049.stdout:(79/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-07-20T17:33:27.333 INFO:teuthology.orchestra.run.smithi049.stdout:(80/131): automake-1.16.1-7.el8.noarch.rpm 5.6 MB/s | 713 kB 00:00 2023-07-20T17:33:27.583 INFO:teuthology.orchestra.run.smithi049.stdout:(81/131): python3-babel-2.5.1-7.el8.noarch.rpm 19 MB/s | 4.8 MB 00:00 2023-07-20T17:33:27.733 INFO:teuthology.orchestra.run.smithi049.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 177 kB/s | 27 kB 00:00 2023-07-20T17:33:27.892 INFO:teuthology.orchestra.run.smithi049.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 2.0 MB/s | 320 kB 00:00 2023-07-20T17:33:28.067 INFO:teuthology.orchestra.run.smithi049.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 251 kB/s | 44 kB 00:00 2023-07-20T17:33:28.267 INFO:teuthology.orchestra.run.smithi049.stdout:(85/131): python3-numpy-1.14.3-10.el8.x86_64.rp 19 MB/s | 3.7 MB 00:00 2023-07-20T17:33:28.401 INFO:teuthology.orchestra.run.smithi049.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 3.4 MB/s | 457 kB 00:00 2023-07-20T17:33:28.551 INFO:teuthology.orchestra.run.smithi049.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.5 MB/s | 225 kB 00:00 2023-07-20T17:33:28.704 INFO:teuthology.orchestra.run.smithi049.stdout:(88/131): uuid-1.6.2-43.el8.x86_64.rpm 420 kB/s | 64 kB 00:00 2023-07-20T17:33:28.818 INFO:teuthology.orchestra.run.smithi049.stdout:(89/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2023-07-20T17:33:28.952 INFO:teuthology.orchestra.run.smithi049.stdout:(90/131): python3-jsonpointer-1.10-11.el8.noarc 150 kB/s | 20 kB 00:00 2023-07-20T17:33:29.094 INFO:teuthology.orchestra.run.smithi049.stdout:(91/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 730 kB/s | 103 kB 00:00 2023-07-20T17:33:29.244 INFO:teuthology.orchestra.run.smithi049.stdout:(92/131): libtool-2.4.6-25.el8.x86_64.rpm 4.6 MB/s | 709 kB 00:00 2023-07-20T17:33:29.436 INFO:teuthology.orchestra.run.smithi049.stdout:(93/131): openblas-threads-0.3.15-3.el8.x86_64. 25 MB/s | 4.9 MB 00:00 2023-07-20T17:33:29.561 INFO:teuthology.orchestra.run.smithi049.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 1.5 MB/s | 198 kB 00:00 2023-07-20T17:33:29.745 INFO:teuthology.orchestra.run.smithi049.stdout:(95/131): openblas-0.3.15-3.el8.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2023-07-20T17:33:29.820 INFO:teuthology.orchestra.run.smithi049.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 2.6 MB/s | 202 kB 00:00 2023-07-20T17:33:29.962 INFO:teuthology.orchestra.run.smithi049.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-07-20T17:33:30.154 INFO:teuthology.orchestra.run.smithi049.stdout:(98/131): postgresql-server-10.21-2.module+el8. 26 MB/s | 5.1 MB 00:00 2023-07-20T17:33:30.280 INFO:teuthology.orchestra.run.smithi049.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 6.3 MB/s | 808 kB 00:00 2023-07-20T17:33:30.388 INFO:teuthology.orchestra.run.smithi049.stdout:(100/131): postgresql-10.21-2.module+el8.6.0+15 14 MB/s | 1.5 MB 00:00 2023-07-20T17:33:30.597 INFO:teuthology.orchestra.run.smithi049.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 224 kB/s | 47 kB 00:00 2023-07-20T17:33:30.730 INFO:teuthology.orchestra.run.smithi049.stdout:(102/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.5 MB/s | 200 kB 00:00 2023-07-20T17:33:30.931 INFO:teuthology.orchestra.run.smithi049.stdout:(103/131): python3-jwt-1.6.1-2.el8.noarch.rpm 213 kB/s | 43 kB 00:00 2023-07-20T17:33:31.006 INFO:teuthology.orchestra.run.smithi049.stdout:(104/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2023-07-20T17:33:31.181 INFO:teuthology.orchestra.run.smithi049.stdout:(105/131): keyutils-libs-devel-1.5.10-9.el8.x86 275 kB/s | 48 kB 00:00 2023-07-20T17:33:31.457 INFO:teuthology.orchestra.run.smithi049.stdout:(106/131): libsepol-devel-2.9-3.el8.x86_64.rpm 316 kB/s | 87 kB 00:00 2023-07-20T17:33:31.590 INFO:teuthology.orchestra.run.smithi049.stdout:(107/131): python3-requests-oauthlib-1.0.0-1.el 320 kB/s | 43 kB 00:00 2023-07-20T17:33:31.690 INFO:teuthology.orchestra.run.smithi049.stdout:(108/131): python3-pycparser-2.14-14.el8.noarch 1.1 MB/s | 109 kB 00:00 2023-07-20T17:33:31.874 INFO:teuthology.orchestra.run.smithi049.stdout:(109/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.2 MB/s | 229 kB 00:00 2023-07-20T17:33:32.024 INFO:teuthology.orchestra.run.smithi049.stdout:(110/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.6 MB/s | 560 kB 00:00 2023-07-20T17:33:32.133 INFO:teuthology.orchestra.run.smithi049.stdout:(111/131): libaio-devel-0.3.112-1.el8.x86_64.rp 173 kB/s | 19 kB 00:00 2023-07-20T17:33:32.291 INFO:teuthology.orchestra.run.smithi049.stdout:(112/131): python3-oauthlib-2.1.0-1.el8.noarch. 979 kB/s | 155 kB 00:00 2023-07-20T17:33:32.417 INFO:teuthology.orchestra.run.smithi049.stdout:(113/131): python3-cryptography-3.2.1-5.el8.x86 4.4 MB/s | 559 kB 00:00 2023-07-20T17:33:32.550 INFO:teuthology.orchestra.run.smithi049.stdout:(114/131): libverto-devel-0.3.0-5.el8.x86_64.rp 134 kB/s | 18 kB 00:00 2023-07-20T17:33:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:(115/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.5 MB/s | 605 kB 00:00 2023-07-20T17:33:32.834 INFO:teuthology.orchestra.run.smithi049.stdout:(116/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.3 MB/s | 238 kB 00:00 2023-07-20T17:33:33.009 INFO:teuthology.orchestra.run.smithi049.stdout:(117/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 13 MB/s | 2.3 MB 00:00 2023-07-20T17:33:33.135 INFO:teuthology.orchestra.run.smithi049.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 578 kB/s | 72 kB 00:00 2023-07-20T17:33:33.243 INFO:teuthology.orchestra.run.smithi049.stdout:(119/131): fuse-2.9.7-15.el8.x86_64.rpm 765 kB/s | 83 kB 00:00 2023-07-20T17:33:33.410 INFO:teuthology.orchestra.run.smithi049.stdout:(120/131): elfutils-libelf-devel-0.186-1.el8.x8 360 kB/s | 60 kB 00:00 2023-07-20T17:33:33.585 INFO:teuthology.orchestra.run.smithi049.stdout:(121/131): libuuid-devel-2.32.1-35.el8.x86_64.r 561 kB/s | 98 kB 00:00 2023-07-20T17:33:33.686 INFO:teuthology.orchestra.run.smithi049.stdout:(122/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 576 kB/s | 58 kB 00:00 2023-07-20T17:33:33.819 INFO:teuthology.orchestra.run.smithi049.stdout:(123/131): libcom_err-devel-1.45.6-4.el8.x86_64 290 kB/s | 39 kB 00:00 2023-07-20T17:33:33.944 INFO:teuthology.orchestra.run.smithi049.stdout:(124/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 5.0 MB/s | 644 kB 00:00 2023-07-20T17:33:34.137 INFO:teuthology.orchestra.run.smithi049.stdout:(125/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 893 kB/s | 171 kB 00:00 2023-07-20T17:33:34.253 INFO:teuthology.orchestra.run.smithi049.stdout:(126/131): NetworkManager-initscripts-updown-1. 1.2 MB/s | 138 kB 00:00 2023-07-20T17:33:36.195 INFO:teuthology.orchestra.run.smithi049.stdout:(127/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 1.9 MB/s | 3.7 MB 00:01 2023-07-20T17:33:36.987 INFO:teuthology.orchestra.run.smithi049.stdout:(128/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 4.6 MB/s | 3.6 MB 00:00 2023-07-20T17:33:37.154 INFO:teuthology.orchestra.run.smithi049.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.7 MB/s | 798 kB 00:00 2023-07-20T17:35:13.762 INFO:teuthology.orchestra.run.smithi049.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-07-20T17:35:14.375 INFO:teuthology.orchestra.run.smithi049.stdout:(130/131): python3-asyncssh-2.7.0-2.el8.noarch. 3.6 kB/s | 476 kB 02:11 2023-07-20T17:35:17.878 INFO:teuthology.orchestra.run.smithi049.stdout:[MIRROR] python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/p/python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-07-20T17:35:18.049 INFO:teuthology.orchestra.run.smithi049.stdout:(131/131): python3-bcrypt-3.1.6-2.el8.1.x86_64. 340 B/s | 44 kB 02:11 2023-07-20T17:35:18.051 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-07-20T17:35:18.051 INFO:teuthology.orchestra.run.smithi049.stdout:Total 777 kB/s | 206 MB 04:31 2023-07-20T17:35:18.851 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-07-20T17:35:18.981 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-07-20T17:35:18.982 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-07-20T17:35:20.159 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-07-20T17:35:20.163 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-07-20T17:35:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-07-20T17:35:25.113 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T17:35:25.278 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T17:35:25.482 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-20T17:35:25.682 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T17:35:25.772 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T17:35:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T17:35:26.434 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T17:35:26.631 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T17:35:27.050 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T17:35:27.248 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T17:35:27.379 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T17:35:27.565 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T17:35:27.665 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T17:35:27.858 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T17:35:28.105 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T17:35:28.310 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T17:35:28.829 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T17:35:29.044 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T17:35:29.181 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T17:35:29.288 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T17:35:29.431 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T17:35:29.516 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T17:35:29.750 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T17:35:29.951 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T17:35:30.114 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T17:35:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T17:35:30.956 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T17:35:31.577 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T17:35:31.764 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T17:35:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T17:35:32.754 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-20T17:35:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-20T17:35:33.193 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-20T17:35:33.387 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-20T17:35:33.624 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-20T17:35:33.852 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T17:35:34.021 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T17:35:34.168 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T17:35:35.102 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T17:35:35.402 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-20T17:35:35.678 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-20T17:35:35.927 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-20T17:35:36.121 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T17:35:37.988 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T17:35:38.292 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-20T17:35:38.510 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T17:35:38.692 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T17:35:38.878 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T17:35:39.033 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T17:35:39.196 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-20T17:35:39.343 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-20T17:35:39.524 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-20T17:35:39.873 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-20T17:35:40.106 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-20T17:35:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-20T17:35:40.742 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-20T17:35:40.949 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-20T17:35:41.161 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-20T17:35:41.427 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-20T17:35:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-20T17:35:41.946 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-20T17:35:42.124 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-20T17:35:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-20T17:35:42.501 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-20T17:35:42.667 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-20T17:35:42.969 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-20T17:35:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-20T17:35:43.487 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-20T17:35:43.703 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-20T17:35:43.870 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-20T17:35:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-20T17:35:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T17:35:44.260 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T17:35:44.507 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-20T17:35:44.726 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-20T17:35:44.879 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T17:35:44.946 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T17:35:45.185 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-20T17:35:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-20T17:35:45.512 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T17:35:45.676 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T17:35:45.915 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T17:35:46.120 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T17:35:46.242 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T17:35:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T17:35:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T17:35:46.809 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T17:35:47.108 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T17:35:47.282 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T17:35:47.426 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T17:35:47.662 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T17:35:47.883 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T17:35:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T17:35:48.298 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T17:35:48.486 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T17:35:48.636 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T17:35:48.829 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T17:35:49.973 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T17:35:50.225 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T17:35:50.454 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T17:35:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T17:35:51.032 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T17:35:51.239 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T17:35:52.059 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T17:35:52.336 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T17:35:52.391 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T17:35:52.561 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T17:35:56.432 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T17:35:56.628 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T17:35:57.249 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T17:35:57.517 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T17:35:57.728 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T17:35:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T17:36:17.191 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T17:36:17.192 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2023-07-20T17:36:17.192 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2023-07-20T17:36:17.192 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /dev 2023-07-20T17:36:17.192 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /run 2023-07-20T17:36:17.192 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2023-07-20T17:36:17.193 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2023-07-20T17:36:17.193 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2023-07-20T17:36:17.193 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2023-07-20T17:36:17.193 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /dev 2023-07-20T17:36:17.193 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:36:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T17:36:19.202 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T17:36:19.356 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T17:36:19.586 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T17:36:19.757 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T17:36:20.238 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T17:36:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T17:36:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T17:36:22.133 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T17:36:22.325 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T17:36:22.446 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T17:36:22.640 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T17:36:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T17:36:23.027 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T17:36:23.322 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T17:36:24.344 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T17:36:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T17:36:24.593 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T17:36:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T17:36:25.179 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T17:36:25.215 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T17:36:25.356 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T17:36:25.367 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T17:36:26.744 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T17:36:26.919 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T17:36:26.953 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T17:36:27.081 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T17:36:27.123 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T17:36:27.264 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T17:36:27.638 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T17:36:27.764 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T17:36:32.617 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T17:36:32.915 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T17:36:33.055 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T17:36:33.268 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T17:36:34.160 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-20T17:36:34.508 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-20T17:36:34.658 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 119/134 2023-07-20T17:36:34.842 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T17:36:35.009 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T17:36:35.147 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T17:36:37.535 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T17:36:37.804 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T17:36:37.821 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T17:36:38.048 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T17:36:38.238 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T17:36:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T17:36:38.773 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T17:36:38.912 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T17:36:38.920 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T17:36:39.096 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-20T17:36:39.287 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T17:36:39.368 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T17:36:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T17:36:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T17:36:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T17:36:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T17:36:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T17:36:39.901 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T17:36:40.103 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T17:36:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T17:36:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T17:36:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T17:36:40.607 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T17:36:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T17:36:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T17:36:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T17:36:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T17:36:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T17:36:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T17:36:40.608 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T17:36:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T17:36:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T17:36:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T17:36:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T17:36:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T17:36:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T17:36:40.609 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T17:36:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T17:36:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T17:36:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T17:36:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T17:36:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T17:36:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T17:36:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T17:36:40.610 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T17:36:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T17:36:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T17:36:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T17:36:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T17:36:40.611 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T17:36:40.619 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T17:36:40.619 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T17:36:40.619 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T17:36:40.619 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T17:36:40.619 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T17:36:40.619 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T17:36:40.620 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T17:36:40.620 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T17:36:40.620 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T17:36:40.620 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T17:36:40.620 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T17:36:40.620 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T17:36:40.620 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T17:36:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T17:36:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T17:36:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T17:36:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T17:36:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T17:36:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T17:36:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T17:36:40.622 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T17:36:40.622 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T17:36:40.623 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T17:36:40.623 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T17:36:40.623 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T17:36:40.623 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T17:36:40.623 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T17:36:40.623 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T17:36:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T17:36:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T17:36:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T17:36:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T17:36:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T17:36:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T17:36:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T17:36:40.625 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T17:36:40.625 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T17:36:40.625 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T17:36:40.625 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T17:36:40.625 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T17:36:40.625 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T17:36:40.626 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-20T17:36:40.626 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-20T17:36:40.626 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-20T17:36:40.626 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-20T17:36:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-20T17:36:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-20T17:36:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-20T17:36:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-20T17:36:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-20T17:36:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-20T17:36:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-20T17:36:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-20T17:36:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-20T17:36:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-20T17:36:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-20T17:36:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-20T17:36:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-20T17:36:40.629 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-20T17:36:40.629 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-20T17:36:40.629 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-20T17:36:40.629 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-20T17:36:40.630 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-20T17:36:40.630 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-20T17:36:40.630 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-20T17:36:40.630 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-20T17:36:40.631 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-20T17:36:40.631 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-20T17:36:40.631 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-20T17:36:40.631 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-20T17:36:40.631 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-20T17:36:40.632 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-20T17:36:40.632 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-20T17:36:40.632 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-20T17:36:40.633 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-20T17:36:40.633 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-20T17:36:40.633 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-20T17:36:40.633 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-20T17:36:40.634 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-20T17:36:40.634 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-20T17:36:40.634 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-20T17:36:40.634 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-20T17:36:40.635 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-20T17:36:40.635 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-20T17:36:40.635 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-20T17:36:40.635 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-20T17:36:40.635 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-20T17:36:40.636 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-20T17:36:40.636 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-20T17:36:40.636 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T17:36:40.636 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-20T17:36:40.637 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T17:36:40.637 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-20T17:36:40.637 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-20T17:36:40.637 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-20T17:36:40.637 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-20T17:36:40.638 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-20T17:36:40.638 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T17:36:40.638 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T17:36:40.638 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T17:36:40.639 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T17:36:40.639 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T17:36:40.639 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T17:36:43.106 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T17:36:43.106 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:06:09 ago on Thu 20 Jul 2023 05:30:34 PM UTC. 2023-07-20T17:36:49.176 INFO:teuthology.orchestra.run.smithi049.stdout:Installed products updated. 2023-07-20T17:36:49.414 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:36:49.415 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2023-07-20T17:36:49.415 INFO:teuthology.orchestra.run.smithi049.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T17:36:49.415 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.415 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.416 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-07-20T17:36:49.416 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T17:36:49.416 INFO:teuthology.orchestra.run.smithi049.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T17:36:49.416 INFO:teuthology.orchestra.run.smithi049.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T17:36:49.416 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.416 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.417 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.417 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.417 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:36:49.417 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.417 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.417 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.417 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:36:49.417 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:36:49.418 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:36:49.418 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:36:49.418 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:36:49.418 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.418 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.418 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:36:49.418 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.418 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.419 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.419 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:36:49.419 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:36:49.419 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.419 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:36:49.419 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T17:36:49.419 INFO:teuthology.orchestra.run.smithi049.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T17:36:49.420 INFO:teuthology.orchestra.run.smithi049.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T17:36:49.420 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T17:36:49.420 INFO:teuthology.orchestra.run.smithi049.stdout: jq-1.6-3.el8.x86_64 2023-07-20T17:36:49.420 INFO:teuthology.orchestra.run.smithi049.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T17:36:49.420 INFO:teuthology.orchestra.run.smithi049.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T17:36:49.420 INFO:teuthology.orchestra.run.smithi049.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T17:36:49.420 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.421 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.421 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.421 INFO:teuthology.orchestra.run.smithi049.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T17:36:49.421 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T17:36:49.421 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T17:36:49.421 INFO:teuthology.orchestra.run.smithi049.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T17:36:49.421 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T17:36:49.422 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T17:36:49.422 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.422 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.422 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T17:36:49.422 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.422 INFO:teuthology.orchestra.run.smithi049.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T17:36:49.423 INFO:teuthology.orchestra.run.smithi049.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T17:36:49.423 INFO:teuthology.orchestra.run.smithi049.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T17:36:49.423 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T17:36:49.423 INFO:teuthology.orchestra.run.smithi049.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T17:36:49.423 INFO:teuthology.orchestra.run.smithi049.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T17:36:49.423 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T17:36:49.423 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T17:36:49.424 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T17:36:49.424 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T17:36:49.424 INFO:teuthology.orchestra.run.smithi049.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T17:36:49.424 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T17:36:49.424 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T17:36:49.424 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T17:36:49.424 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T17:36:49.425 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T17:36:49.425 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T17:36:49.425 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T17:36:49.425 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T17:36:49.425 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T17:36:49.426 INFO:teuthology.orchestra.run.smithi049.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T17:36:49.426 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T17:36:49.426 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.426 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.426 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.426 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T17:36:49.426 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T17:36:49.427 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T17:36:49.427 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T17:36:49.427 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T17:36:49.427 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T17:36:49.427 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T17:36:49.427 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T17:36:49.427 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T17:36:49.427 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T17:36:49.428 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T17:36:49.428 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T17:36:49.428 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T17:36:49.428 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T17:36:49.428 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T17:36:49.428 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T17:36:49.429 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T17:36:49.429 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T17:36:49.429 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T17:36:49.429 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T17:36:49.429 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T17:36:49.429 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T17:36:49.429 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T17:36:49.429 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T17:36:49.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T17:36:49.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T17:36:49.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T17:36:49.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T17:36:49.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T17:36:49.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.431 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.431 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T17:36:49.431 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T17:36:49.431 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.431 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T17:36:49.431 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T17:36:49.431 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T17:36:49.432 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T17:36:49.432 INFO:teuthology.orchestra.run.smithi049.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T17:36:49.432 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T17:36:49.432 INFO:teuthology.orchestra.run.smithi049.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T17:36:49.432 INFO:teuthology.orchestra.run.smithi049.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T17:36:49.432 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T17:36:49.432 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T17:36:49.433 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T17:36:49.433 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T17:36:49.433 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T17:36:49.433 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.433 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.433 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:36:49.433 INFO:teuthology.orchestra.run.smithi049.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T17:36:49.434 INFO:teuthology.orchestra.run.smithi049.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T17:36:49.434 INFO:teuthology.orchestra.run.smithi049.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T17:36:49.434 INFO:teuthology.orchestra.run.smithi049.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T17:36:49.434 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T17:36:49.435 INFO:teuthology.orchestra.run.smithi049.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T17:36:49.435 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:36:49.435 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-07-20T17:36:49.605 DEBUG:teuthology.parallel:result is None 2023-07-20T17:37:38.855 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 35 kB/s | 16 MB 07:30 2023-07-20T17:37:42.287 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 432 kB/s | 24 kB 00:00 2023-07-20T17:37:43.597 INFO:teuthology.orchestra.run.smithi079.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 MB/s | 47 MB 00:01 2023-07-20T17:37:51.673 INFO:teuthology.orchestra.run.smithi079.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 MB/s | 53 MB 00:01 2023-07-20T17:38:02.802 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T17:38:02.803 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T17:38:02.817 INFO:teuthology.orchestra.run.smithi079.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T17:38:02.818 INFO:teuthology.orchestra.run.smithi079.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T17:38:02.819 INFO:teuthology.orchestra.run.smithi079.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T17:38:02.820 INFO:teuthology.orchestra.run.smithi079.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T17:38:02.822 INFO:teuthology.orchestra.run.smithi079.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T17:38:02.825 INFO:teuthology.orchestra.run.smithi079.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T17:38:02.826 INFO:teuthology.orchestra.run.smithi079.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T17:38:02.957 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-07-20T17:38:02.969 INFO:teuthology.orchestra.run.smithi079.stdout:=========================================================================================================================== 2023-07-20T17:38:02.969 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-07-20T17:38:02.970 INFO:teuthology.orchestra.run.smithi079.stdout:=========================================================================================================================== 2023-07-20T17:38:02.970 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-07-20T17:38:02.970 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T17:38:02.970 INFO:teuthology.orchestra.run.smithi079.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T17:38:02.970 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T17:38:02.970 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T17:38:02.970 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T17:38:02.971 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T17:38:02.971 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T17:38:02.971 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T17:38:02.971 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T17:38:02.971 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T17:38:02.971 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T17:38:02.971 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T17:38:02.972 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T17:38:02.972 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T17:38:02.972 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T17:38:02.972 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T17:38:02.972 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T17:38:02.972 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T17:38:02.972 INFO:teuthology.orchestra.run.smithi079.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T17:38:02.973 INFO:teuthology.orchestra.run.smithi079.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T17:38:02.973 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T17:38:02.973 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T17:38:02.973 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T17:38:02.973 INFO:teuthology.orchestra.run.smithi079.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T17:38:02.973 INFO:teuthology.orchestra.run.smithi079.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T17:38:02.973 INFO:teuthology.orchestra.run.smithi079.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T17:38:02.973 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T17:38:02.974 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T17:38:02.974 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T17:38:02.974 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T17:38:02.974 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T17:38:02.974 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T17:38:02.974 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T17:38:02.974 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T17:38:02.975 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T17:38:02.975 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T17:38:02.975 INFO:teuthology.orchestra.run.smithi079.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T17:38:02.975 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2023-07-20T17:38:02.975 INFO:teuthology.orchestra.run.smithi079.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T17:38:02.975 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T17:38:02.975 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T17:38:02.975 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2023-07-20T17:38:02.976 INFO:teuthology.orchestra.run.smithi079.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T17:38:02.976 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T17:38:02.976 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T17:38:02.976 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T17:38:02.976 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T17:38:02.976 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T17:38:02.976 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T17:38:02.977 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T17:38:02.977 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T17:38:02.977 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T17:38:02.977 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T17:38:02.977 INFO:teuthology.orchestra.run.smithi079.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T17:38:02.978 INFO:teuthology.orchestra.run.smithi079.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T17:38:02.978 INFO:teuthology.orchestra.run.smithi079.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T17:38:02.978 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T17:38:02.978 INFO:teuthology.orchestra.run.smithi079.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T17:38:02.979 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T17:38:02.979 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T17:38:02.979 INFO:teuthology.orchestra.run.smithi079.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T17:38:02.979 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T17:38:02.980 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T17:38:02.980 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T17:38:02.980 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T17:38:02.980 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T17:38:02.980 INFO:teuthology.orchestra.run.smithi079.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T17:38:02.981 INFO:teuthology.orchestra.run.smithi079.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T17:38:02.981 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T17:38:02.981 INFO:teuthology.orchestra.run.smithi079.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T17:38:02.982 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T17:38:02.982 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T17:38:02.982 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T17:38:02.982 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T17:38:02.983 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T17:38:02.983 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T17:38:02.983 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T17:38:02.984 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T17:38:02.984 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T17:38:02.984 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T17:38:02.984 INFO:teuthology.orchestra.run.smithi079.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T17:38:02.985 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T17:38:02.985 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T17:38:02.985 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T17:38:02.985 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T17:38:02.986 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T17:38:02.986 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T17:38:02.986 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T17:38:02.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T17:38:02.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T17:38:02.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T17:38:02.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T17:38:02.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T17:38:02.987 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T17:38:02.988 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T17:38:02.988 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T17:38:02.988 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T17:38:02.988 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T17:38:02.988 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T17:38:02.988 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T17:38:02.988 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T17:38:02.989 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T17:38:02.989 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T17:38:02.989 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T17:38:02.989 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T17:38:02.989 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T17:38:02.989 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T17:38:02.989 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T17:38:02.989 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T17:38:02.990 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T17:38:02.990 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T17:38:02.990 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T17:38:02.990 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T17:38:02.990 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T17:38:02.990 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T17:38:02.990 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T17:38:02.990 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T17:38:02.991 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T17:38:02.991 INFO:teuthology.orchestra.run.smithi079.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T17:38:02.991 INFO:teuthology.orchestra.run.smithi079.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T17:38:02.991 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T17:38:02.992 INFO:teuthology.orchestra.run.smithi079.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T17:38:02.992 INFO:teuthology.orchestra.run.smithi079.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T17:38:02.992 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T17:38:02.992 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T17:38:02.993 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T17:38:02.993 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T17:38:02.993 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T17:38:02.993 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T17:38:02.994 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T17:38:02.994 INFO:teuthology.orchestra.run.smithi079.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T17:38:02.994 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T17:38:02.994 INFO:teuthology.orchestra.run.smithi079.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T17:38:02.995 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling module streams: 2023-07-20T17:38:02.995 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql 10 2023-07-20T17:38:02.995 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:38:02.995 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-07-20T17:38:02.996 INFO:teuthology.orchestra.run.smithi079.stdout:=========================================================================================================================== 2023-07-20T17:38:02.996 INFO:teuthology.orchestra.run.smithi079.stdout:Install 128 Packages 2023-07-20T17:38:02.996 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 3 Packages 2023-07-20T17:38:02.997 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:38:02.998 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 206 M 2023-07-20T17:38:02.998 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-07-20T17:38:03.472 INFO:teuthology.orchestra.run.smithi079.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 23 kB/s | 6.4 kB 00:00 2023-07-20T17:38:04.539 INFO:teuthology.orchestra.run.smithi079.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 797 kB/s | 850 kB 00:01 2023-07-20T17:38:04.739 INFO:teuthology.orchestra.run.smithi079.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 836 kB/s | 167 kB 00:00 2023-07-20T17:38:05.131 INFO:teuthology.orchestra.run.smithi079.stdout:(4/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 2.8 MB/s | 5.4 MB 00:01 2023-07-20T17:38:05.573 INFO:teuthology.orchestra.run.smithi079.stdout:(5/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 3.5 MB/s | 1.6 MB 00:00 2023-07-20T17:38:07.565 INFO:teuthology.orchestra.run.smithi079.stdout:(6/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 2.3 MB/s | 4.6 MB 00:01 2023-07-20T17:38:08.573 INFO:teuthology.orchestra.run.smithi079.stdout:(7/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 611 kB/s | 2.3 MB 00:03 2023-07-20T17:38:17.347 INFO:teuthology.orchestra.run.smithi079.stdout:(8/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x86 1.8 MB/s | 18 MB 00:09 2023-07-20T17:38:17.455 INFO:teuthology.orchestra.run.smithi079.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 225 kB/s | 24 kB 00:00 2023-07-20T17:38:23.096 INFO:teuthology.orchestra.run.smithi079.stdout:(10/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 955 kB/s | 14 MB 00:14 2023-07-20T17:38:23.380 INFO:teuthology.orchestra.run.smithi079.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 919 kB/s | 260 kB 00:00 2023-07-20T17:38:23.822 INFO:teuthology.orchestra.run.smithi079.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 70 kB/s | 31 kB 00:00 2023-07-20T17:38:24.339 INFO:teuthology.orchestra.run.smithi079.stdout:(13/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8 1.0 MB/s | 22 MB 00:21 2023-07-20T17:38:24.931 INFO:teuthology.orchestra.run.smithi079.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 235 kB/s | 139 kB 00:00 2023-07-20T17:38:25.014 INFO:teuthology.orchestra.run.smithi079.stdout:(15/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 647 kB/s | 771 kB 00:01 2023-07-20T17:38:25.323 INFO:teuthology.orchestra.run.smithi079.stdout:(16/131): librados-devel-18.0.0-5040.g1b0b2eb0. 355 kB/s | 139 kB 00:00 2023-07-20T17:38:25.606 INFO:teuthology.orchestra.run.smithi079.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 798 kB/s | 472 kB 00:00 2023-07-20T17:38:25.690 INFO:teuthology.orchestra.run.smithi079.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 537 kB/s | 45 kB 00:00 2023-07-20T17:38:25.907 INFO:teuthology.orchestra.run.smithi079.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 566 kB/s | 122 kB 00:00 2023-07-20T17:38:26.182 INFO:teuthology.orchestra.run.smithi079.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 716 kB/s | 197 kB 00:00 2023-07-20T17:38:26.557 INFO:teuthology.orchestra.run.smithi079.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 988 kB/s | 370 kB 00:00 2023-07-20T17:38:26.966 INFO:teuthology.orchestra.run.smithi079.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 888 kB/s | 363 kB 00:00 2023-07-20T17:38:27.100 INFO:teuthology.orchestra.run.smithi079.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 800 kB/s | 107 kB 00:00 2023-07-20T17:38:27.258 INFO:teuthology.orchestra.run.smithi079.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 593 kB/s | 94 kB 00:00 2023-07-20T17:38:32.241 INFO:teuthology.orchestra.run.smithi079.stdout:(25/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 726 kB/s | 3.5 MB 00:04 2023-07-20T17:38:32.524 INFO:teuthology.orchestra.run.smithi079.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 631 kB/s | 179 kB 00:00 2023-07-20T17:38:32.641 INFO:teuthology.orchestra.run.smithi079.stdout:(27/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 948 kB/s | 6.8 MB 00:07 2023-07-20T17:38:32.658 INFO:teuthology.orchestra.run.smithi079.stdout:(28/131): ceph-grafana-dashboards-18.0.0-5040.g 172 kB/s | 23 kB 00:00 2023-07-20T17:38:32.817 INFO:teuthology.orchestra.run.smithi079.stdout:(29/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 739 kB/s | 129 kB 00:00 2023-07-20T17:38:34.734 INFO:teuthology.orchestra.run.smithi079.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 847 kB/s | 1.7 MB 00:02 2023-07-20T17:38:35.126 INFO:teuthology.orchestra.run.smithi079.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 615 kB/s | 241 kB 00:00 2023-07-20T17:38:35.217 INFO:teuthology.orchestra.run.smithi079.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 517 kB/s | 47 kB 00:00 2023-07-20T17:38:35.301 INFO:teuthology.orchestra.run.smithi079.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 175 kB/s | 15 kB 00:00 2023-07-20T17:38:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 659 kB/s | 258 kB 00:00 2023-07-20T17:38:35.951 INFO:teuthology.orchestra.run.smithi079.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 640 kB/s | 165 kB 00:00 2023-07-20T17:38:36.043 INFO:teuthology.orchestra.run.smithi079.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 196 kB/s | 18 kB 00:00 2023-07-20T17:38:38.135 INFO:teuthology.orchestra.run.smithi079.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 146 kB/s | 306 kB 00:02 2023-07-20T17:38:38.344 INFO:teuthology.orchestra.run.smithi079.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 282 kB/s | 59 kB 00:00 2023-07-20T17:38:38.577 INFO:teuthology.orchestra.run.smithi079.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 322 kB/s | 75 kB 00:00 2023-07-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi079.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 539 kB/s | 476 kB 00:00 2023-07-20T17:38:39.586 INFO:teuthology.orchestra.run.smithi079.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 349 kB/s | 44 kB 00:00 2023-07-20T17:38:39.945 INFO:teuthology.orchestra.run.smithi079.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 517 kB/s | 185 kB 00:00 2023-07-20T17:38:40.087 INFO:teuthology.orchestra.run.smithi079.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 233 kB/s | 33 kB 00:00 2023-07-20T17:38:40.245 INFO:teuthology.orchestra.run.smithi079.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 103 kB/s | 16 kB 00:00 2023-07-20T17:38:40.596 INFO:teuthology.orchestra.run.smithi079.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 493 kB/s | 173 kB 00:00 2023-07-20T17:38:41.004 INFO:teuthology.orchestra.run.smithi079.stdout:(46/131): ceph-mgr-diskprediction-local-18.0.0- 927 kB/s | 7.4 MB 00:08 2023-07-20T17:38:41.471 INFO:teuthology.orchestra.run.smithi079.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 439 kB/s | 384 kB 00:00 2023-07-20T17:38:41.630 INFO:teuthology.orchestra.run.smithi079.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 70 kB/s | 11 kB 00:00 2023-07-20T17:38:41.755 INFO:teuthology.orchestra.run.smithi079.stdout:(49/131): python3-google-auth-1.1.1-10.el8.noar 108 kB/s | 81 kB 00:00 2023-07-20T17:38:41.780 INFO:teuthology.orchestra.run.smithi079.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 117 kB/s | 18 kB 00:00 2023-07-20T17:38:41.914 INFO:teuthology.orchestra.run.smithi079.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 366 kB/s | 49 kB 00:00 2023-07-20T17:38:42.064 INFO:teuthology.orchestra.run.smithi079.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 391 kB/s | 59 kB 00:00 2023-07-20T17:38:42.214 INFO:teuthology.orchestra.run.smithi079.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 402 kB/s | 60 kB 00:00 2023-07-20T17:38:42.823 INFO:teuthology.orchestra.run.smithi079.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 465 kB/s | 283 kB 00:00 2023-07-20T17:38:42.990 INFO:teuthology.orchestra.run.smithi079.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 93 kB/s | 16 kB 00:00 2023-07-20T17:38:43.155 INFO:teuthology.orchestra.run.smithi079.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 204 kB/s | 34 kB 00:00 2023-07-20T17:38:43.632 INFO:teuthology.orchestra.run.smithi079.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 412 kB/s | 196 kB 00:00 2023-07-20T17:38:43.782 INFO:teuthology.orchestra.run.smithi079.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 410 kB/s | 61 kB 00:00 2023-07-20T17:38:43.932 INFO:teuthology.orchestra.run.smithi079.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 125 kB/s | 19 kB 00:00 2023-07-20T17:38:44.074 INFO:teuthology.orchestra.run.smithi079.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 169 kB/s | 24 kB 00:00 2023-07-20T17:38:44.233 INFO:teuthology.orchestra.run.smithi079.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 185 kB/s | 29 kB 00:00 2023-07-20T17:38:44.383 INFO:teuthology.orchestra.run.smithi079.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 183 kB/s | 27 kB 00:00 2023-07-20T17:38:44.984 INFO:teuthology.orchestra.run.smithi079.stdout:(63/131): python3-kubernetes-11.0.0-6.el8.noarc 370 kB/s | 1.2 MB 00:03 2023-07-20T17:38:45.484 INFO:teuthology.orchestra.run.smithi079.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 502 kB/s | 251 kB 00:00 2023-07-20T17:38:45.642 INFO:teuthology.orchestra.run.smithi079.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 203 kB/s | 255 kB 00:01 2023-07-20T17:38:45.659 INFO:teuthology.orchestra.run.smithi079.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 350 kB/s | 61 kB 00:00 2023-07-20T17:38:45.784 INFO:teuthology.orchestra.run.smithi079.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 184 kB/s | 23 kB 00:00 2023-07-20T17:38:45.910 INFO:teuthology.orchestra.run.smithi079.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 319 kB/s | 85 kB 00:00 2023-07-20T17:38:45.935 INFO:teuthology.orchestra.run.smithi079.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T17:38:46.311 INFO:teuthology.orchestra.run.smithi079.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 37 MB/s | 14 MB 00:00 2023-07-20T17:38:46.419 INFO:teuthology.orchestra.run.smithi079.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.0 MB/s | 110 kB 00:00 2023-07-20T17:38:46.553 INFO:teuthology.orchestra.run.smithi079.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 406 kB/s | 54 kB 00:00 2023-07-20T17:38:46.720 INFO:teuthology.orchestra.run.smithi079.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2023-07-20T17:38:46.845 INFO:teuthology.orchestra.run.smithi079.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2023-07-20T17:38:46.987 INFO:teuthology.orchestra.run.smithi079.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 274 kB/s | 39 kB 00:00 2023-07-20T17:38:47.154 INFO:teuthology.orchestra.run.smithi079.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 4.0 MB/s | 688 kB 00:00 2023-07-20T17:38:47.337 INFO:teuthology.orchestra.run.smithi079.stdout:(77/131): python3-jinja2-2.10.1-3.el8.noarch.rp 2.9 MB/s | 538 kB 00:00 2023-07-20T17:38:47.446 INFO:teuthology.orchestra.run.smithi079.stdout:(78/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2023-07-20T17:38:47.655 INFO:teuthology.orchestra.run.smithi079.stdout:(79/131): oniguruma-6.8.2-2.el8.x86_64.rpm 900 kB/s | 187 kB 00:00 2023-07-20T17:38:47.855 INFO:teuthology.orchestra.run.smithi079.stdout:(80/131): automake-1.16.1-7.el8.noarch.rpm 3.5 MB/s | 713 kB 00:00 2023-07-20T17:38:48.080 INFO:teuthology.orchestra.run.smithi079.stdout:(81/131): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2023-07-20T17:38:48.131 INFO:teuthology.orchestra.run.smithi079.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 533 kB/s | 27 kB 00:00 2023-07-20T17:38:48.247 INFO:teuthology.orchestra.run.smithi079.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 2.7 MB/s | 320 kB 00:00 2023-07-20T17:38:48.423 INFO:teuthology.orchestra.run.smithi079.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 251 kB/s | 44 kB 00:00 2023-07-20T17:38:48.656 INFO:teuthology.orchestra.run.smithi079.stdout:(85/131): python3-numpy-1.14.3-10.el8.x86_64.rp 16 MB/s | 3.7 MB 00:00 2023-07-20T17:38:48.790 INFO:teuthology.orchestra.run.smithi079.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 3.4 MB/s | 457 kB 00:00 2023-07-20T17:38:48.922 INFO:teuthology.orchestra.run.smithi079.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.7 MB/s | 225 kB 00:00 2023-07-20T17:38:49.124 INFO:teuthology.orchestra.run.smithi079.stdout:(88/131): uuid-1.6.2-43.el8.x86_64.rpm 318 kB/s | 64 kB 00:00 2023-07-20T17:38:49.249 INFO:teuthology.orchestra.run.smithi079.stdout:(89/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2023-07-20T17:38:49.383 INFO:teuthology.orchestra.run.smithi079.stdout:(90/131): python3-jsonpointer-1.10-11.el8.noarc 150 kB/s | 20 kB 00:00 2023-07-20T17:38:49.508 INFO:teuthology.orchestra.run.smithi079.stdout:(91/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 828 kB/s | 103 kB 00:00 2023-07-20T17:38:49.650 INFO:teuthology.orchestra.run.smithi079.stdout:(92/131): libtool-2.4.6-25.el8.x86_64.rpm 4.9 MB/s | 709 kB 00:00 2023-07-20T17:38:49.825 INFO:teuthology.orchestra.run.smithi079.stdout:(93/131): openblas-threads-0.3.15-3.el8.x86_64. 28 MB/s | 4.9 MB 00:00 2023-07-20T17:38:49.942 INFO:teuthology.orchestra.run.smithi079.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 1.7 MB/s | 198 kB 00:00 2023-07-20T17:38:50.168 INFO:teuthology.orchestra.run.smithi079.stdout:(95/131): openblas-0.3.15-3.el8.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2023-07-20T17:38:50.326 INFO:teuthology.orchestra.run.smithi079.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2023-07-20T17:38:50.427 INFO:teuthology.orchestra.run.smithi079.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.5 MB/s | 354 kB 00:00 2023-07-20T17:38:50.685 INFO:teuthology.orchestra.run.smithi079.stdout:(98/131): postgresql-server-10.21-2.module+el8. 20 MB/s | 5.1 MB 00:00 2023-07-20T17:38:50.886 INFO:teuthology.orchestra.run.smithi079.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 4.0 MB/s | 808 kB 00:00 2023-07-20T17:38:51.002 INFO:teuthology.orchestra.run.smithi079.stdout:(100/131): postgresql-10.21-2.module+el8.6.0+15 13 MB/s | 1.5 MB 00:00 2023-07-20T17:38:51.120 INFO:teuthology.orchestra.run.smithi079.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 402 kB/s | 47 kB 00:00 2023-07-20T17:38:51.293 INFO:teuthology.orchestra.run.smithi079.stdout:(102/131): thrift-0.13.0-2.el8.x86_64.rpm 322 kB/s | 1.7 MB 00:05 2023-07-20T17:38:51.328 INFO:teuthology.orchestra.run.smithi079.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 959 kB/s | 200 kB 00:00 2023-07-20T17:38:51.462 INFO:teuthology.orchestra.run.smithi079.stdout:(104/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.7 MB/s | 220 kB 00:00 2023-07-20T17:38:51.487 INFO:teuthology.orchestra.run.smithi079.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 219 kB/s | 43 kB 00:00 2023-07-20T17:38:51.604 INFO:teuthology.orchestra.run.smithi079.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 339 kB/s | 48 kB 00:00 2023-07-20T17:38:51.630 INFO:teuthology.orchestra.run.smithi079.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 613 kB/s | 87 kB 00:00 2023-07-20T17:38:51.705 INFO:teuthology.orchestra.run.smithi079.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 425 kB/s | 43 kB 00:00 2023-07-20T17:38:51.763 INFO:teuthology.orchestra.run.smithi079.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 818 kB/s | 109 kB 00:00 2023-07-20T17:38:51.822 INFO:teuthology.orchestra.run.smithi079.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.9 MB/s | 229 kB 00:00 2023-07-20T17:38:51.922 INFO:teuthology.orchestra.run.smithi079.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.4 MB/s | 560 kB 00:00 2023-07-20T17:38:51.989 INFO:teuthology.orchestra.run.smithi079.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 112 kB/s | 19 kB 00:00 2023-07-20T17:38:52.056 INFO:teuthology.orchestra.run.smithi079.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.1 MB/s | 155 kB 00:00 2023-07-20T17:38:52.106 INFO:teuthology.orchestra.run.smithi079.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 4.7 MB/s | 559 kB 00:00 2023-07-20T17:38:52.173 INFO:teuthology.orchestra.run.smithi079.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 153 kB/s | 18 kB 00:00 2023-07-20T17:38:52.466 INFO:teuthology.orchestra.run.smithi079.stdout:(116/131): pcre2-devel-10.32-2.el8.x86_64.rpm 1.6 MB/s | 605 kB 00:00 2023-07-20T17:38:52.499 INFO:teuthology.orchestra.run.smithi079.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 730 kB/s | 238 kB 00:00 2023-07-20T17:38:52.591 INFO:teuthology.orchestra.run.smithi079.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 789 kB/s | 72 kB 00:00 2023-07-20T17:38:52.666 INFO:teuthology.orchestra.run.smithi079.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 12 MB/s | 2.3 MB 00:00 2023-07-20T17:38:52.691 INFO:teuthology.orchestra.run.smithi079.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 825 kB/s | 83 kB 00:00 2023-07-20T17:38:52.733 INFO:teuthology.orchestra.run.smithi079.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 898 kB/s | 60 kB 00:00 2023-07-20T17:38:52.800 INFO:teuthology.orchestra.run.smithi079.stdout:(122/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 865 kB/s | 58 kB 00:00 2023-07-20T17:38:52.826 INFO:teuthology.orchestra.run.smithi079.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 732 kB/s | 98 kB 00:00 2023-07-20T17:38:52.934 INFO:teuthology.orchestra.run.smithi079.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 289 kB/s | 39 kB 00:00 2023-07-20T17:38:53.043 INFO:teuthology.orchestra.run.smithi079.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 2.9 MB/s | 644 kB 00:00 2023-07-20T17:38:53.127 INFO:teuthology.orchestra.run.smithi079.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 891 kB/s | 171 kB 00:00 2023-07-20T17:38:53.152 INFO:teuthology.orchestra.run.smithi079.stdout:(127/131): NetworkManager-initscripts-updown-1. 1.2 MB/s | 138 kB 00:00 2023-07-20T17:38:56.060 INFO:teuthology.orchestra.run.smithi079.stdout:(128/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 1.3 MB/s | 3.7 MB 00:02 2023-07-20T17:38:56.194 INFO:teuthology.orchestra.run.smithi079.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 5.9 MB/s | 798 kB 00:00 2023-07-20T17:38:57.394 INFO:teuthology.orchestra.run.smithi079.stdout:(130/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 876 kB/s | 3.6 MB 00:04 2023-07-20T17:39:05.076 INFO:teuthology.orchestra.run.smithi079.stdout:(131/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8. 1.0 MB/s | 48 MB 00:47 2023-07-20T17:39:05.078 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-07-20T17:39:05.079 INFO:teuthology.orchestra.run.smithi079.stdout:Total 3.3 MB/s | 206 MB 01:02 2023-07-20T17:39:05.898 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-07-20T17:39:06.018 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-07-20T17:39:06.019 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-07-20T17:39:07.148 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-07-20T17:39:07.152 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-07-20T17:39:12.037 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-07-20T17:39:12.259 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T17:39:12.428 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T17:39:12.643 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-20T17:39:12.861 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T17:39:12.959 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T17:39:13.278 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T17:39:13.804 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T17:39:14.044 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T17:39:14.445 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T17:39:14.636 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T17:39:14.767 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T17:39:14.944 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T17:39:15.058 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T17:39:15.237 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T17:39:15.456 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T17:39:15.680 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T17:39:16.071 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T17:39:16.339 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T17:39:16.486 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T17:39:16.586 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T17:39:16.735 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T17:39:16.844 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T17:39:17.002 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T17:39:17.310 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T17:39:17.951 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T17:39:18.135 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T17:39:18.715 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T17:39:18.893 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T17:39:19.593 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T17:39:19.874 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-20T17:39:20.121 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-20T17:39:20.371 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-20T17:39:20.582 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-20T17:39:20.789 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-20T17:39:21.069 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T17:39:21.241 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T17:39:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T17:39:22.303 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T17:39:22.607 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-20T17:39:22.877 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-20T17:39:23.136 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-20T17:39:23.359 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T17:39:25.129 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T17:39:25.502 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-20T17:39:25.711 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T17:39:25.910 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T17:39:26.114 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T17:39:26.287 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T17:39:26.449 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-20T17:39:26.659 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-20T17:39:26.844 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-20T17:39:26.995 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-20T17:39:27.232 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-20T17:39:27.423 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-20T17:39:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-20T17:39:28.053 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-20T17:39:28.226 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-20T17:39:28.469 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-20T17:39:28.704 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-20T17:39:29.040 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-20T17:39:29.249 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-20T17:39:29.455 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-20T17:39:29.712 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-20T17:39:29.903 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-20T17:39:30.102 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-20T17:39:30.432 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-20T17:39:30.737 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-20T17:39:30.995 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-20T17:39:31.230 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-20T17:39:31.431 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-20T17:39:31.583 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-20T17:39:31.757 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T17:39:31.860 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T17:39:32.101 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-20T17:39:32.294 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-20T17:39:32.482 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T17:39:32.563 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T17:39:32.772 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-20T17:39:32.984 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-20T17:39:33.141 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T17:39:33.313 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T17:39:33.606 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T17:39:33.847 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T17:39:33.995 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T17:39:34.193 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T17:39:34.400 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T17:39:34.603 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T17:39:34.827 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T17:39:34.993 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T17:39:35.130 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T17:39:35.398 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T17:39:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T17:39:35.846 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T17:39:36.079 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T17:39:36.289 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T17:39:36.505 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T17:39:36.731 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T17:39:37.824 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T17:39:38.101 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T17:39:38.414 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T17:39:38.733 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T17:39:39.058 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T17:39:39.250 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T17:39:40.089 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T17:39:40.247 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T17:39:40.306 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T17:39:40.506 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T17:39:44.203 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T17:39:44.730 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T17:39:45.327 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T17:39:45.612 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T17:39:45.732 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T17:39:45.904 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T17:40:04.795 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T17:40:04.795 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2023-07-20T17:40:04.796 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2023-07-20T17:40:04.796 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /dev 2023-07-20T17:40:04.796 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /run 2023-07-20T17:40:04.796 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2023-07-20T17:40:04.796 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2023-07-20T17:40:04.796 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2023-07-20T17:40:04.796 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2023-07-20T17:40:04.797 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /dev 2023-07-20T17:40:04.797 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:40:05.184 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T17:40:06.765 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T17:40:06.974 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T17:40:07.192 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T17:40:07.375 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T17:40:07.827 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T17:40:07.975 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T17:40:07.992 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T17:40:09.621 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T17:40:09.818 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T17:40:09.947 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T17:40:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T17:40:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T17:40:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T17:40:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T17:40:11.834 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T17:40:11.990 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T17:40:12.080 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T17:40:12.484 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T17:40:12.667 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T17:40:12.703 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T17:40:12.848 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T17:40:12.860 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T17:40:14.191 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T17:40:14.388 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T17:40:14.422 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T17:40:14.623 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T17:40:14.666 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T17:40:14.832 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T17:40:15.184 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T17:40:15.348 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T17:40:19.901 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T17:40:20.212 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T17:40:20.405 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T17:40:20.601 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T17:40:21.478 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-20T17:40:21.828 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-20T17:40:22.016 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 119/134 2023-07-20T17:40:22.217 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T17:40:22.376 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T17:40:22.523 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T17:40:24.677 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T17:40:24.930 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T17:40:24.948 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T17:40:25.146 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T17:40:25.404 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T17:40:25.727 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T17:40:26.006 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T17:40:26.196 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T17:40:26.204 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T17:40:26.393 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-20T17:40:26.579 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T17:40:26.663 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T17:40:26.904 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T17:40:26.916 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T17:40:26.916 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T17:40:27.098 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T17:40:27.099 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T17:40:27.376 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T17:40:27.594 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T17:40:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T17:40:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T17:40:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T17:40:28.074 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T17:40:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T17:40:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T17:40:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T17:40:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T17:40:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T17:40:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T17:40:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T17:40:28.075 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T17:40:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T17:40:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T17:40:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T17:40:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T17:40:28.076 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T17:40:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T17:40:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T17:40:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T17:40:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T17:40:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T17:40:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T17:40:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T17:40:28.077 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T17:40:28.078 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T17:40:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T17:40:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T17:40:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T17:40:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T17:40:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T17:40:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T17:40:28.085 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T17:40:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T17:40:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T17:40:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T17:40:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T17:40:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T17:40:28.086 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T17:40:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T17:40:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T17:40:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T17:40:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T17:40:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T17:40:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T17:40:28.087 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T17:40:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T17:40:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T17:40:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T17:40:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T17:40:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T17:40:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T17:40:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T17:40:28.088 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T17:40:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T17:40:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T17:40:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T17:40:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T17:40:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T17:40:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T17:40:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T17:40:28.089 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T17:40:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T17:40:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T17:40:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T17:40:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T17:40:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T17:40:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T17:40:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T17:40:28.090 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T17:40:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T17:40:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T17:40:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-20T17:40:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-20T17:40:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-20T17:40:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-20T17:40:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-20T17:40:28.091 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-20T17:40:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-20T17:40:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-20T17:40:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-20T17:40:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-20T17:40:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-20T17:40:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-20T17:40:28.092 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-20T17:40:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-20T17:40:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-20T17:40:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-20T17:40:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-20T17:40:28.093 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-20T17:40:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-20T17:40:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-20T17:40:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-20T17:40:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-20T17:40:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-20T17:40:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-20T17:40:28.094 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-20T17:40:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-20T17:40:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-20T17:40:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-20T17:40:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-20T17:40:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-20T17:40:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-20T17:40:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-20T17:40:28.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-20T17:40:28.096 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-20T17:40:28.096 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-20T17:40:28.096 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-20T17:40:28.096 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-20T17:40:28.096 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-20T17:40:28.096 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-20T17:40:28.096 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-20T17:40:28.097 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-20T17:40:28.097 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-20T17:40:28.097 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-20T17:40:28.097 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-20T17:40:28.097 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-20T17:40:28.097 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-20T17:40:28.097 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-20T17:40:28.098 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-20T17:40:28.098 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T17:40:28.098 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-20T17:40:28.098 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T17:40:28.098 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-20T17:40:28.098 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-20T17:40:28.098 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-20T17:40:28.098 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-20T17:40:28.099 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-20T17:40:28.099 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T17:40:28.099 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T17:40:28.099 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T17:40:28.099 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T17:40:28.099 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T17:40:28.099 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T17:40:30.592 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T17:40:30.593 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:39 ago on Thu 20 Jul 2023 05:37:51 PM UTC. 2023-07-20T17:40:37.792 INFO:teuthology.orchestra.run.smithi079.stdout:Installed products updated. 2023-07-20T17:40:38.005 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:40:38.006 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2023-07-20T17:40:38.006 INFO:teuthology.orchestra.run.smithi079.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T17:40:38.006 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.006 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.006 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-07-20T17:40:38.006 INFO:teuthology.orchestra.run.smithi079.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T17:40:38.007 INFO:teuthology.orchestra.run.smithi079.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T17:40:38.007 INFO:teuthology.orchestra.run.smithi079.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T17:40:38.007 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.007 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.007 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.007 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.008 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:40:38.008 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.008 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.008 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.008 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:40:38.008 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:40:38.008 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:40:38.009 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:40:38.009 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:40:38.009 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.009 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.009 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:40:38.010 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.010 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.010 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.010 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:40:38.010 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:40:38.010 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.010 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T17:40:38.010 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T17:40:38.011 INFO:teuthology.orchestra.run.smithi079.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T17:40:38.011 INFO:teuthology.orchestra.run.smithi079.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T17:40:38.011 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T17:40:38.011 INFO:teuthology.orchestra.run.smithi079.stdout: jq-1.6-3.el8.x86_64 2023-07-20T17:40:38.011 INFO:teuthology.orchestra.run.smithi079.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T17:40:38.011 INFO:teuthology.orchestra.run.smithi079.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T17:40:38.011 INFO:teuthology.orchestra.run.smithi079.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T17:40:38.011 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.012 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.012 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.012 INFO:teuthology.orchestra.run.smithi079.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T17:40:38.012 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T17:40:38.012 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T17:40:38.012 INFO:teuthology.orchestra.run.smithi079.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T17:40:38.012 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T17:40:38.013 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T17:40:38.013 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.013 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.013 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T17:40:38.013 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.013 INFO:teuthology.orchestra.run.smithi079.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T17:40:38.013 INFO:teuthology.orchestra.run.smithi079.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T17:40:38.013 INFO:teuthology.orchestra.run.smithi079.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T17:40:38.014 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T17:40:38.014 INFO:teuthology.orchestra.run.smithi079.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T17:40:38.014 INFO:teuthology.orchestra.run.smithi079.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T17:40:38.014 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T17:40:38.014 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T17:40:38.014 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T17:40:38.014 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T17:40:38.014 INFO:teuthology.orchestra.run.smithi079.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T17:40:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T17:40:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T17:40:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T17:40:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T17:40:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T17:40:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T17:40:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T17:40:38.016 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T17:40:38.016 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T17:40:38.016 INFO:teuthology.orchestra.run.smithi079.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T17:40:38.016 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T17:40:38.016 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.016 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.016 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.016 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T17:40:38.017 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T17:40:38.017 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T17:40:38.017 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T17:40:38.017 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T17:40:38.017 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T17:40:38.017 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T17:40:38.017 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T17:40:38.017 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T17:40:38.018 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T17:40:38.018 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T17:40:38.018 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T17:40:38.018 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T17:40:38.018 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T17:40:38.018 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T17:40:38.018 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T17:40:38.018 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T17:40:38.019 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T17:40:38.019 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T17:40:38.019 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T17:40:38.019 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T17:40:38.019 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T17:40:38.019 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T17:40:38.020 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T17:40:38.020 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T17:40:38.020 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T17:40:38.020 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T17:40:38.020 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T17:40:38.020 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T17:40:38.020 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.021 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.021 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T17:40:38.021 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T17:40:38.021 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.021 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T17:40:38.021 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T17:40:38.021 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T17:40:38.022 INFO:teuthology.orchestra.run.smithi079.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T17:40:38.022 INFO:teuthology.orchestra.run.smithi079.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T17:40:38.022 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T17:40:38.022 INFO:teuthology.orchestra.run.smithi079.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T17:40:38.022 INFO:teuthology.orchestra.run.smithi079.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T17:40:38.022 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T17:40:38.022 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T17:40:38.023 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T17:40:38.023 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T17:40:38.023 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T17:40:38.023 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.023 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.023 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T17:40:38.023 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T17:40:38.023 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T17:40:38.024 INFO:teuthology.orchestra.run.smithi079.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T17:40:38.024 INFO:teuthology.orchestra.run.smithi079.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T17:40:38.024 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T17:40:38.024 INFO:teuthology.orchestra.run.smithi079.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T17:40:38.024 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-07-20T17:40:38.025 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-07-20T17:40:38.199 DEBUG:teuthology.parallel:result is None 2023-07-20T17:40:38.200 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T17:40:39.042 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T17:40:39.072 INFO:teuthology.orchestra.run.smithi049.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T17:40:39.073 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T17:40:39.073 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T17:40:39.074 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T17:40:39.753 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T17:40:39.780 INFO:teuthology.orchestra.run.smithi079.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T17:40:39.781 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T17:40:39.781 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T17:40:39.782 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T17:40:40.462 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T17:40:40.490 INFO:teuthology.orchestra.run.smithi097.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T17:40:40.491 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T17:40:40.491 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T17:40:40.491 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-20T17:40:40.492 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:40:40.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T17:40:40.541 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-07-20T17:40:40.541 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T17:40:40.588 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-07-20T17:40:40.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T17:40:40.635 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-20T17:40:40.636 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:40:40.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T17:40:40.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T17:40:40.736 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-07-20T17:40:40.737 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T17:40:40.768 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T17:40:40.838 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-07-20T17:40:40.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T17:40:40.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T17:40:40.942 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-20T17:40:40.943 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:40:40.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T17:40:40.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T17:40:41.043 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-07-20T17:40:41.043 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T17:40:41.074 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T17:40:41.144 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-07-20T17:40:41.144 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T17:40:41.175 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T17:40:41.245 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-20T17:40:41.246 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:40:41.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T17:40:41.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T17:40:41.346 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-07-20T17:40:41.346 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T17:40:41.378 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T17:40:41.447 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-07-20T17:40:41.447 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T17:40:41.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T17:40:41.547 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-20T17:40:41.642 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-20T17:40:41.642 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T17:40:41.651 INFO:tasks.cephadm:Cluster fsid is 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:40:41.651 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-20T17:40:41.652 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.49', 'mon.b': '172.21.15.79', 'mon.c': '172.21.15.97'} 2023-07-20T17:40:41.652 INFO:tasks.cephadm:First mon is mon.a on smithi049 2023-07-20T17:40:41.653 INFO:tasks.cephadm:First mgr is x 2023-07-20T17:40:41.653 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-20T17:40:41.653 DEBUG:teuthology.orchestra.run.smithi049:> sudo hostname $(hostname -s) 2023-07-20T17:40:41.687 DEBUG:teuthology.orchestra.run.smithi079:> sudo hostname $(hostname -s) 2023-07-20T17:40:41.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo hostname $(hostname -s) 2023-07-20T17:40:41.750 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-20T17:40:41.750 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T17:40:42.386 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71562/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-5040-g1b0b2eb0', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-5040.g1b0b2eb0'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-19 22:37:13.930734', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230719.201220', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-07-20T17:40:42.977 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-pdonnell-testing-20230719.201220, sha1 1b0b2eb005a7c51d97a788b9900d511e1394b537 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T17:40:42.977 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm 2023-07-20T17:40:42.978 DEBUG:teuthology.orchestra.run.smithi049:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T17:40:44.714 INFO:teuthology.orchestra.run.smithi049.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 17:40 /home/ubuntu/cephtest/cephadm 2023-07-20T17:40:44.715 DEBUG:teuthology.orchestra.run.smithi079:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T17:40:46.314 INFO:teuthology.orchestra.run.smithi079.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 17:40 /home/ubuntu/cephtest/cephadm 2023-07-20T17:40:46.315 DEBUG:teuthology.orchestra.run.smithi097:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T17:40:48.109 INFO:teuthology.orchestra.run.smithi097.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 17:40 /home/ubuntu/cephtest/cephadm 2023-07-20T17:40:48.110 DEBUG:teuthology.orchestra.run.smithi049:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T17:40:48.133 DEBUG:teuthology.orchestra.run.smithi079:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T17:40:48.154 DEBUG:teuthology.orchestra.run.smithi097:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T17:40:48.193 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 on all hosts... 2023-07-20T17:40:48.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T17:40:48.195 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T17:40:48.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T17:40:49.456 INFO:teuthology.orchestra.run.smithi049.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T17:40:49.593 INFO:teuthology.orchestra.run.smithi097.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T17:40:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T17:41:14.337 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2023-07-20T17:41:14.338 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T17:41:14.338 INFO:teuthology.orchestra.run.smithi049.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T17:41:14.338 INFO:teuthology.orchestra.run.smithi049.stdout: "repo_digests": [ 2023-07-20T17:41:14.338 INFO:teuthology.orchestra.run.smithi049.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T17:41:14.338 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2023-07-20T17:41:14.338 INFO:teuthology.orchestra.run.smithi049.stdout:} 2023-07-20T17:41:43.695 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2023-07-20T17:41:43.696 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T17:41:43.696 INFO:teuthology.orchestra.run.smithi079.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T17:41:43.696 INFO:teuthology.orchestra.run.smithi079.stdout: "repo_digests": [ 2023-07-20T17:41:43.696 INFO:teuthology.orchestra.run.smithi079.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T17:41:43.697 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2023-07-20T17:41:43.697 INFO:teuthology.orchestra.run.smithi079.stdout:} 2023-07-20T17:41:49.026 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-07-20T17:41:49.026 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T17:41:49.026 INFO:teuthology.orchestra.run.smithi097.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T17:41:49.026 INFO:teuthology.orchestra.run.smithi097.stdout: "repo_digests": [ 2023-07-20T17:41:49.026 INFO:teuthology.orchestra.run.smithi097.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T17:41:49.027 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2023-07-20T17:41:49.027 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-07-20T17:41:49.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph 2023-07-20T17:41:49.085 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph 2023-07-20T17:41:49.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph 2023-07-20T17:41:49.154 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /etc/ceph 2023-07-20T17:41:49.186 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /etc/ceph 2023-07-20T17:41:49.216 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /etc/ceph 2023-07-20T17:41:49.242 INFO:tasks.cephadm:Writing seed config... 2023-07-20T17:41:49.244 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-20T17:41:49.244 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-20T17:41:49.244 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-20T17:41:49.245 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-20T17:41:49.245 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-20T17:41:49.245 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-07-20T17:41:49.245 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-20T17:41:49.245 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-20T17:41:49.246 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-20T17:41:49.246 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-20T17:41:49.246 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-20T17:41:49.246 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-20T17:41:49.246 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-20T17:41:49.247 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-20T17:41:49.247 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-20T17:41:49.247 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-20T17:41:49.247 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-20T17:41:49.247 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-20T17:41:49.248 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-20T17:41:49.248 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-20T17:41:49.248 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-20T17:41:49.248 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-07-20T17:41:49.248 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-20T17:41:49.249 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-20T17:41:49.249 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-20T17:41:49.249 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-20T17:41:49.249 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-20T17:41:49.250 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-20T17:41:49.250 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-20T17:41:49.250 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-07-20T17:41:49.251 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-20T17:41:49.251 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-07-20T17:41:49.251 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-07-20T17:41:49.252 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-20T17:41:49.252 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-20T17:41:49.253 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-07-20T17:41:49.253 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-20T17:41:49.253 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-20T17:41:49.254 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-20T17:41:49.254 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-20T17:41:49.254 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-20T17:41:49.255 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-20T17:41:49.255 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-20T17:41:49.256 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-07-20T17:41:49.257 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:41:49.258 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-20T17:41:49.278 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8c050cea-2724-11ee-9b34-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-20T17:41:49.278 DEBUG:teuthology.orchestra.run.smithi049:mon.a> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.a.service 2023-07-20T17:41:49.320 DEBUG:teuthology.orchestra.run.smithi049:mgr.x> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.x.service 2023-07-20T17:41:49.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:-- Logs begin at Thu 2023-07-20 17:16:49 UTC. -- 2023-07-20T17:41:49.350 INFO:tasks.cephadm:Bootstrapping... 2023-07-20T17:41:49.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 -v bootstrap --fsid 8c050cea-2724-11ee-9b34-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.49 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-20T17:41:49.379 INFO:journalctl@ceph.mgr.x.smithi049.stdout:-- Logs begin at Thu 2023-07-20 17:16:49 UTC. -- 2023-07-20T17:41:49.520 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-07-20T17:41:49.521 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537', '-v', 'bootstrap', '--fsid', '8c050cea-2724-11ee-9b34-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.49', '--skip-admin-label'] 2023-07-20T17:41:49.671 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 4.1.1 2023-07-20T17:41:49.671 INFO:teuthology.orchestra.run.smithi049.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-20T17:41:49.671 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying podman|docker is present... 2023-07-20T17:41:49.814 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 4.1.1 2023-07-20T17:41:49.814 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying lvm2 is present... 2023-07-20T17:41:49.814 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying time synchronization is in place... 2023-07-20T17:41:49.821 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T17:41:49.821 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T17:41:49.826 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T17:41:49.827 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2023-07-20T17:41:49.833 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout enabled 2023-07-20T17:41:49.838 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout active 2023-07-20T17:41:49.839 INFO:teuthology.orchestra.run.smithi049.stdout:Unit chronyd.service is enabled and running 2023-07-20T17:41:49.839 INFO:teuthology.orchestra.run.smithi049.stdout:Repeating the final host check... 2023-07-20T17:41:49.984 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 4.1.1 2023-07-20T17:41:49.984 INFO:teuthology.orchestra.run.smithi049.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-20T17:41:49.984 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl is present 2023-07-20T17:41:49.984 INFO:teuthology.orchestra.run.smithi049.stdout:lvcreate is present 2023-07-20T17:41:49.990 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T17:41:49.990 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T17:41:49.996 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T17:41:49.996 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2023-07-20T17:41:50.002 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout enabled 2023-07-20T17:41:50.007 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout active 2023-07-20T17:41:50.008 INFO:teuthology.orchestra.run.smithi049.stdout:Unit chronyd.service is enabled and running 2023-07-20T17:41:50.008 INFO:teuthology.orchestra.run.smithi049.stdout:Host looks OK 2023-07-20T17:41:50.008 INFO:teuthology.orchestra.run.smithi049.stdout:Cluster fsid: 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:41:50.008 INFO:teuthology.orchestra.run.smithi049.stdout:Acquiring lock 140483030653416 on /run/cephadm/8c050cea-2724-11ee-9b34-001a4aab830c.lock 2023-07-20T17:41:50.008 INFO:teuthology.orchestra.run.smithi049.stdout:Lock 140483030653416 acquired on /run/cephadm/8c050cea-2724-11ee-9b34-001a4aab830c.lock 2023-07-20T17:41:50.009 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying IP 172.21.15.49 port 3300 ... 2023-07-20T17:41:50.009 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying IP 172.21.15.49 port 6789 ... 2023-07-20T17:41:50.009 INFO:teuthology.orchestra.run.smithi049.stdout:Base mon IP(s) is [172.21.15.49:3300, 172.21.15.49:6789], mon addrv is [v2:172.21.15.49:3300,v1:172.21.15.49:6789] 2023-07-20T17:41:50.013 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-07-20T17:41:50.013 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.49 metric 100 2023-07-20T17:41:50.015 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-20T17:41:50.015 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-07-20T17:41:50.015 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1793sec hoplimit 64 pref medium 2023-07-20T17:41:50.017 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-20T17:41:50.018 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-20T17:41:50.018 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T17:41:50.018 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-07-20T17:41:50.018 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1566/64 scope link 2023-07-20T17:41:50.018 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T17:41:50.019 INFO:teuthology.orchestra.run.smithi049.stdout:Mon IP `172.21.15.49` is in CIDR network `172.21.0.0/20` 2023-07-20T17:41:50.019 INFO:teuthology.orchestra.run.smithi049.stdout:Mon IP `172.21.15.49` is in CIDR network `172.21.0.0/20` 2023-07-20T17:41:50.019 INFO:teuthology.orchestra.run.smithi049.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-20T17:41:50.019 INFO:teuthology.orchestra.run.smithi049.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-20T17:41:50.020 INFO:teuthology.orchestra.run.smithi049.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T17:41:50.565 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T17:41:50.565 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T17:41:50.566 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Getting image source signatures 2023-07-20T17:41:50.566 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying blob sha256:9a3af2d1f966fade87cf08b715899d7eff2dde16dabc6e2e4c3ec2059da8adc1 2023-07-20T17:41:50.566 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-20T17:41:50.566 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying config sha256:3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T17:41:50.566 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-20T17:41:50.566 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Storing signatures 2023-07-20T17:41:51.713 INFO:teuthology.orchestra.run.smithi049.stdout:ceph: stdout ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T17:41:51.713 INFO:teuthology.orchestra.run.smithi049.stdout:Ceph version: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T17:41:51.713 INFO:teuthology.orchestra.run.smithi049.stdout:Extracting ceph user uid/gid from container image... 2023-07-20T17:41:52.464 INFO:teuthology.orchestra.run.smithi049.stdout:stat: stdout 167 167 2023-07-20T17:41:52.464 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial keys... 2023-07-20T17:41:53.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQDgcblkB08qMBAABQkjvElVrYOoZvweZvGZyQ== 2023-07-20T17:41:53.980 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQDhcblksed3JBAARggS6qSAreeLvdI5g3bnOw== 2023-07-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQDicblkWnkFFBAAMlDbGEllDj6KSTjdGdf15w== 2023-07-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial monmap... 2023-07-20T17:41:55.478 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T17:41:55.478 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-20T17:41:55.479 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:41:55.479 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T17:41:55.479 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool for a [v2:172.21.15.49:3300,v1:172.21.15.49:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T17:41:55.479 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = pacific 2023-07-20T17:41:55.479 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: set fsid to 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:41:55.479 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T17:41:55.479 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:41:55.480 INFO:teuthology.orchestra.run.smithi049.stdout:Creating mon... 2023-07-20T17:41:56.244 INFO:teuthology.orchestra.run.smithi049.stdout:create mon.a on 2023-07-20T17:41:56.400 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-20T17:41:56.494 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-20T17:41:56.618 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8c050cea-2724-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-8c050cea-2724-11ee-9b34-001a4aab830c.target. 2023-07-20T17:41:56.619 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8c050cea-2724-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-8c050cea-2724-11ee-9b34-001a4aab830c.target. 2023-07-20T17:41:56.804 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.a 2023-07-20T17:41:56.804 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.a.service: Unit ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.a.service not loaded. 2023-07-20T17:41:56.896 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8c050cea-2724-11ee-9b34-001a4aab830c.target.wants/ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-8c050cea-2724-11ee-9b34-001a4aab830c@.service. 2023-07-20T17:41:57.206 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:56 smithi049 systemd[1]: Starting Ceph mon.a for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:41:57.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:57 smithi049 podman[95988]: 2023-07-20T17:41:57.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:57 smithi049 bash[95988]: 198c3eff8578f9c9181ed3601bbd613d45e59b9a4c77b88c7a4450c497382c5d 2023-07-20T17:41:57.504 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T17:41:57.505 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2023-07-20T17:41:57.512 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T17:41:57.512 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2023-07-20T17:41:57.512 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2023-07-20T17:41:57.512 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T17:41:57.513 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mon to start... 2023-07-20T17:41:57.513 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mon... 2023-07-20T17:41:57.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:57 smithi049 ceph-mon[96028]: mkfs 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:41:57.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:57 smithi049 ceph-mon[96028]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T17:41:57.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:57 smithi049 systemd[1]: Started Ceph mon.a for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:41:58.565 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout cluster: 2023-07-20T17:41:58.565 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout id: 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:41:58.565 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-20T17:41:58.565 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:41:58.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout services: 2023-07-20T17:41:58.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.729096s) 2023-07-20T17:41:58.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-20T17:41:58.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-20T17:41:58.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:41:58.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout data: 2023-07-20T17:41:58.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-20T17:41:58.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-20T17:41:58.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-20T17:41:58.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout pgs: 2023-07-20T17:41:58.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:41:58.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.054+0000 7fd9789c8700 1 Processor -- start 2023-07-20T17:41:58.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.054+0000 7fd9789c8700 1 -- start start 2023-07-20T17:41:58.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.054+0000 7fd9789c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd974106390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:41:58.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.054+0000 7fd9789c8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd974106960 con 0x7fd974105f70 2023-07-20T17:41:58.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.055+0000 7fd97259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd974106390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:41:58.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.055+0000 7fd97259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd974106390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35128/0 (socket says 172.21.15.49:35128) 2023-07-20T17:41:58.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.055+0000 7fd97259c700 1 -- 172.21.15.49:0/26513769 learned_addr learned my addr 172.21.15.49:0/26513769 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:41:58.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.055+0000 7fd97259c700 1 -- 172.21.15.49:0/26513769 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974107170 con 0x7fd974105f70 2023-07-20T17:41:58.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.055+0000 7fd97259c700 1 --2- 172.21.15.49:0/26513769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd974106390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd9640098a0 tx=0x7fd964009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94db2b192eab03 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:41:58.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.055+0000 7fd97159a700 1 -- 172.21.15.49:0/26513769 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd964004160 con 0x7fd974105f70 2023-07-20T17:41:58.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.055+0000 7fd97159a700 1 -- 172.21.15.49:0/26513769 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9640042c0 con 0x7fd974105f70 2023-07-20T17:41:58.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.055+0000 7fd97159a700 1 -- 172.21.15.49:0/26513769 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9640045a0 con 0x7fd974105f70 2023-07-20T17:41:58.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.056+0000 7fd9789c8700 1 -- 172.21.15.49:0/26513769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 msgr2=0x7fd974106390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:41:58.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.056+0000 7fd9789c8700 1 --2- 172.21.15.49:0/26513769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd974106390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd9640098a0 tx=0x7fd964009bb0 comp rx=0 tx=0).stop 2023-07-20T17:41:58.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.056+0000 7fd9789c8700 1 -- 172.21.15.49:0/26513769 shutdown_connections 2023-07-20T17:41:58.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.056+0000 7fd9789c8700 1 --2- 172.21.15.49:0/26513769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd974106390 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:41:58.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.056+0000 7fd9789c8700 1 -- 172.21.15.49:0/26513769 >> 172.21.15.49:0/26513769 conn(0x7fd974101550 msgr2=0x7fd974103970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:41:58.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.056+0000 7fd9789c8700 1 -- 172.21.15.49:0/26513769 shutdown_connections 2023-07-20T17:41:58.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.056+0000 7fd9789c8700 1 -- 172.21.15.49:0/26513769 wait complete. 2023-07-20T17:41:58.570 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.057+0000 7fd9789c8700 1 Processor -- start 2023-07-20T17:41:58.570 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.057+0000 7fd9789c8700 1 -- start start 2023-07-20T17:41:58.570 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.057+0000 7fd9789c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd9741993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:41:58.570 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.057+0000 7fd9789c8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd974199920 con 0x7fd974105f70 2023-07-20T17:41:58.570 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.057+0000 7fd97259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd9741993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:41:58.570 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.057+0000 7fd97259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd9741993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35130/0 (socket says 172.21.15.49:35130) 2023-07-20T17:41:58.570 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.057+0000 7fd97259c700 1 -- 172.21.15.49:0/1998764345 learned_addr learned my addr 172.21.15.49:0/1998764345 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:41:58.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.058+0000 7fd97259c700 1 -- 172.21.15.49:0/1998764345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd964009550 con 0x7fd974105f70 2023-07-20T17:41:58.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.058+0000 7fd97259c700 1 --2- 172.21.15.49:0/1998764345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd9741993e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd964004000 tx=0x7fd964004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:41:58.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.058+0000 7fd9637fe700 1 -- 172.21.15.49:0/1998764345 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd964011630 con 0x7fd974105f70 2023-07-20T17:41:58.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.058+0000 7fd9637fe700 1 -- 172.21.15.49:0/1998764345 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd964011790 con 0x7fd974105f70 2023-07-20T17:41:58.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.058+0000 7fd9789c8700 1 -- 172.21.15.49:0/1998764345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd974199b20 con 0x7fd974105f70 2023-07-20T17:41:58.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.058+0000 7fd9637fe700 1 -- 172.21.15.49:0/1998764345 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd964011a30 con 0x7fd974105f70 2023-07-20T17:41:58.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.058+0000 7fd9789c8700 1 -- 172.21.15.49:0/1998764345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd974199f40 con 0x7fd974105f70 2023-07-20T17:41:58.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.058+0000 7fd9637fe700 1 -- 172.21.15.49:0/1998764345 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd964022070 con 0x7fd974105f70 2023-07-20T17:41:58.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.058+0000 7fd9637fe700 1 -- 172.21.15.49:0/1998764345 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd96401c450 con 0x7fd974105f70 2023-07-20T17:41:58.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.059+0000 7fd9789c8700 1 -- 172.21.15.49:0/1998764345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd974192c90 con 0x7fd974105f70 2023-07-20T17:41:58.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.061+0000 7fd9637fe700 1 -- 172.21.15.49:0/1998764345 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fd96401b450 con 0x7fd974105f70 2023-07-20T17:41:58.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.194+0000 7fd9789c8700 1 -- 172.21.15.49:0/1998764345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fd97402c970 con 0x7fd974105f70 2023-07-20T17:41:58.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.194+0000 7fd9637fe700 1 -- 172.21.15.49:0/1998764345 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fd96401b5f0 con 0x7fd974105f70 2023-07-20T17:41:58.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.195+0000 7fd9789c8700 1 -- 172.21.15.49:0/1998764345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 msgr2=0x7fd9741993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:41:58.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.195+0000 7fd9789c8700 1 --2- 172.21.15.49:0/1998764345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd9741993e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd964004000 tx=0x7fd964004750 comp rx=0 tx=0).stop 2023-07-20T17:41:58.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.195+0000 7fd9789c8700 1 -- 172.21.15.49:0/1998764345 shutdown_connections 2023-07-20T17:41:58.573 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.195+0000 7fd9789c8700 1 --2- 172.21.15.49:0/1998764345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd974105f70 0x7fd9741993e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:41:58.573 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.195+0000 7fd9789c8700 1 -- 172.21.15.49:0/1998764345 >> 172.21.15.49:0/1998764345 conn(0x7fd974101550 msgr2=0x7fd97418d930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:41:58.573 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.195+0000 7fd9789c8700 1 -- 172.21.15.49:0/1998764345 shutdown_connections 2023-07-20T17:41:58.573 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:58.195+0000 7fd9789c8700 1 -- 172.21.15.49:0/1998764345 wait complete. 2023-07-20T17:41:58.573 INFO:teuthology.orchestra.run.smithi049.stdout:mon is available 2023-07-20T17:41:58.573 INFO:teuthology.orchestra.run.smithi049.stdout:Assimilating anything we can from ceph.conf... 2023-07-20T17:41:58.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:58 smithi049 ceph-mon[96028]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T17:41:58.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:58 smithi049 ceph-mon[96028]: monmap e1: 1 mons at {a=[v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0]} removed_ranks: {} 2023-07-20T17:41:58.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:58 smithi049 ceph-mon[96028]: fsmap 2023-07-20T17:41:58.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:58 smithi049 ceph-mon[96028]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T17:41:58.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:58 smithi049 ceph-mon[96028]: mgrmap e1: no daemons active 2023-07-20T17:41:58.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:58 smithi049 ceph-mon[96028]: from='client.? 172.21.15.49:0/1998764345' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-20T17:41:59.487 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:59 smithi049 ceph-mon[96028]: from='client.? 172.21.15.49:0/70993074' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T17:41:59.487 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:41:59 smithi049 ceph-mon[96028]: from='client.? 172.21.15.49:0/70993074' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-20T17:41:59.711 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:41:59.711 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T17:41:59.711 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout fsid = 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:41:59.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.49:3300,v1:172.21.15.49:6789] 2023-07-20T17:41:59.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T17:41:59.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T17:41:59.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T17:41:59.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T17:41:59.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:41:59.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T17:41:59.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T17:41:59.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:41:59.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T17:41:59.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T17:41:59.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T17:41:59.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T17:41:59.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T17:41:59.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T17:41:59.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.073+0000 7f3f3188b700 1 Processor -- start 2023-07-20T17:41:59.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.073+0000 7f3f3188b700 1 -- start start 2023-07-20T17:41:59.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.073+0000 7f3f3188b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c104650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:41:59.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.073+0000 7f3f3188b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c104c20 con 0x7f3f2c104230 2023-07-20T17:41:59.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.073+0000 7f3f2affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c104650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:41:59.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.073+0000 7f3f2affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c104650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35132/0 (socket says 172.21.15.49:35132) 2023-07-20T17:41:59.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.073+0000 7f3f2affd700 1 -- 172.21.15.49:0/485048039 learned_addr learned my addr 172.21.15.49:0/485048039 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:41:59.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.074+0000 7f3f2affd700 1 -- 172.21.15.49:0/485048039 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c1071c0 con 0x7f3f2c104230 2023-07-20T17:41:59.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.074+0000 7f3f2affd700 1 --2- 172.21.15.49:0/485048039 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c104650 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0098a0 tx=0x7f3f1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb3ad8cfbc2b74c server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:41:59.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.074+0000 7f3f29ffb700 1 -- 172.21.15.49:0/485048039 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f1c004160 con 0x7f3f2c104230 2023-07-20T17:41:59.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.074+0000 7f3f29ffb700 1 -- 172.21.15.49:0/485048039 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1c0042c0 con 0x7f3f2c104230 2023-07-20T17:41:59.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.074+0000 7f3f29ffb700 1 -- 172.21.15.49:0/485048039 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f1c0045a0 con 0x7f3f2c104230 2023-07-20T17:41:59.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.074+0000 7f3f3188b700 1 -- 172.21.15.49:0/485048039 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 msgr2=0x7f3f2c104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:41:59.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.074+0000 7f3f3188b700 1 --2- 172.21.15.49:0/485048039 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c104650 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0098a0 tx=0x7f3f1c009bb0 comp rx=0 tx=0).stop 2023-07-20T17:41:59.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.075+0000 7f3f3188b700 1 -- 172.21.15.49:0/485048039 shutdown_connections 2023-07-20T17:41:59.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.075+0000 7f3f3188b700 1 --2- 172.21.15.49:0/485048039 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c104650 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:41:59.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.075+0000 7f3f3188b700 1 -- 172.21.15.49:0/485048039 >> 172.21.15.49:0/485048039 conn(0x7f3f2c0ffbc0 msgr2=0x7f3f2c101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:41:59.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.075+0000 7f3f3188b700 1 -- 172.21.15.49:0/485048039 shutdown_connections 2023-07-20T17:41:59.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.075+0000 7f3f3188b700 1 -- 172.21.15.49:0/485048039 wait complete. 2023-07-20T17:41:59.717 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.075+0000 7f3f3188b700 1 Processor -- start 2023-07-20T17:41:59.717 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.075+0000 7f3f3188b700 1 -- start start 2023-07-20T17:41:59.717 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f3188b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:41:59.717 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f3188b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c199940 con 0x7f3f2c104230 2023-07-20T17:41:59.717 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f2affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:41:59.717 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f2affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c199400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35134/0 (socket says 172.21.15.49:35134) 2023-07-20T17:41:59.717 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f2affd700 1 -- 172.21.15.49:0/70993074 learned_addr learned my addr 172.21.15.49:0/70993074 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:41:59.718 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f2affd700 1 -- 172.21.15.49:0/70993074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c009550 con 0x7f3f2c104230 2023-07-20T17:41:59.718 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f2affd700 1 --2- 172.21.15.49:0/70993074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c199400 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c004000 tx=0x7f3f1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:41:59.718 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f30889700 1 -- 172.21.15.49:0/70993074 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f1c011630 con 0x7f3f2c104230 2023-07-20T17:41:59.718 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f30889700 1 -- 172.21.15.49:0/70993074 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1c011790 con 0x7f3f2c104230 2023-07-20T17:41:59.718 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f3188b700 1 -- 172.21.15.49:0/70993074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2c199b40 con 0x7f3f2c104230 2023-07-20T17:41:59.718 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f30889700 1 -- 172.21.15.49:0/70993074 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f1c011a30 con 0x7f3f2c104230 2023-07-20T17:41:59.718 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.076+0000 7f3f3188b700 1 -- 172.21.15.49:0/70993074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2c199f60 con 0x7f3f2c104230 2023-07-20T17:41:59.719 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.077+0000 7f3f30889700 1 -- 172.21.15.49:0/70993074 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3f1c022070 con 0x7f3f2c104230 2023-07-20T17:41:59.719 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.077+0000 7f3f30889700 1 -- 172.21.15.49:0/70993074 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3f1c01c450 con 0x7f3f2c104230 2023-07-20T17:41:59.719 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.078+0000 7f3f3188b700 1 -- 172.21.15.49:0/70993074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2c192ce0 con 0x7f3f2c104230 2023-07-20T17:41:59.719 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.080+0000 7f3f30889700 1 -- 172.21.15.49:0/70993074 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f3f1c01b450 con 0x7f3f2c104230 2023-07-20T17:41:59.719 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.207+0000 7f3f3188b700 1 -- 172.21.15.49:0/70993074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3f2c02cba0 con 0x7f3f2c104230 2023-07-20T17:41:59.719 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.210+0000 7f3f30889700 1 -- 172.21.15.49:0/70993074 <== mon.0 v2:172.21.15.49:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f1c01b5f0 con 0x7f3f2c104230 2023-07-20T17:41:59.719 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.210+0000 7f3f30889700 1 -- 172.21.15.49:0/70993074 <== mon.0 v2:172.21.15.49:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f3f1c024630 con 0x7f3f2c104230 2023-07-20T17:41:59.719 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.211+0000 7f3f3188b700 1 -- 172.21.15.49:0/70993074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 msgr2=0x7f3f2c199400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:41:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.211+0000 7f3f3188b700 1 --2- 172.21.15.49:0/70993074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c199400 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c004000 tx=0x7f3f1c004750 comp rx=0 tx=0).stop 2023-07-20T17:41:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.211+0000 7f3f3188b700 1 -- 172.21.15.49:0/70993074 shutdown_connections 2023-07-20T17:41:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.211+0000 7f3f3188b700 1 --2- 172.21.15.49:0/70993074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2c104230 0x7f3f2c199400 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:41:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.211+0000 7f3f3188b700 1 -- 172.21.15.49:0/70993074 >> 172.21.15.49:0/70993074 conn(0x7f3f2c0ffbc0 msgr2=0x7f3f2c18d980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:41:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.212+0000 7f3f3188b700 1 -- 172.21.15.49:0/70993074 shutdown_connections 2023-07-20T17:41:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:41:59.212+0000 7f3f3188b700 1 -- 172.21.15.49:0/70993074 wait complete. 2023-07-20T17:41:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:Generating new minimal ceph.conf... 2023-07-20T17:42:00.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:00 smithi049 ceph-mon[96028]: from='client.? 172.21.15.49:0/1863943033' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:42:00.755 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.193+0000 7fcb4dffb700 1 Processor -- start 2023-07-20T17:42:00.755 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.193+0000 7fcb4dffb700 1 -- start start 2023-07-20T17:42:00.755 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.194+0000 7fcb4dffb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb48108e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:00.756 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.194+0000 7fcb4dffb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb48079df0 con 0x7fcb48106a90 2023-07-20T17:42:00.756 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.194+0000 7fcb477fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb48108e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:00.756 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.194+0000 7fcb477fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb48108e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35136/0 (socket says 172.21.15.49:35136) 2023-07-20T17:42:00.756 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.194+0000 7fcb477fe700 1 -- 172.21.15.49:0/1055506310 learned_addr learned my addr 172.21.15.49:0/1055506310 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:00.756 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.194+0000 7fcb477fe700 1 -- 172.21.15.49:0/1055506310 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48079f30 con 0x7fcb48106a90 2023-07-20T17:42:00.756 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.194+0000 7fcb477fe700 1 --2- 172.21.15.49:0/1055506310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb48108e80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcb380098a0 tx=0x7fcb38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f90cfbcba620d1 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:00.756 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.194+0000 7fcb467fc700 1 -- 172.21.15.49:0/1055506310 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb38004160 con 0x7fcb48106a90 2023-07-20T17:42:00.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.195+0000 7fcb467fc700 1 -- 172.21.15.49:0/1055506310 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb380042c0 con 0x7fcb48106a90 2023-07-20T17:42:00.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.195+0000 7fcb467fc700 1 -- 172.21.15.49:0/1055506310 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb38011a00 con 0x7fcb48106a90 2023-07-20T17:42:00.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.195+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1055506310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 msgr2=0x7fcb48108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:00.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.195+0000 7fcb4dffb700 1 --2- 172.21.15.49:0/1055506310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb48108e80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcb380098a0 tx=0x7fcb38009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:00.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.195+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1055506310 shutdown_connections 2023-07-20T17:42:00.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.195+0000 7fcb4dffb700 1 --2- 172.21.15.49:0/1055506310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb48108e80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:00.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.195+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1055506310 >> 172.21.15.49:0/1055506310 conn(0x7fcb48100400 msgr2=0x7fcb48102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:00.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.195+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1055506310 shutdown_connections 2023-07-20T17:42:00.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.195+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1055506310 wait complete. 2023-07-20T17:42:00.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.196+0000 7fcb4dffb700 1 Processor -- start 2023-07-20T17:42:00.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.196+0000 7fcb4dffb700 1 -- start start 2023-07-20T17:42:00.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.196+0000 7fcb4dffb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb481063f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:00.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.196+0000 7fcb4dffb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb48106930 con 0x7fcb48106a90 2023-07-20T17:42:00.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.196+0000 7fcb477fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb481063f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:00.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.196+0000 7fcb477fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb481063f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35138/0 (socket says 172.21.15.49:35138) 2023-07-20T17:42:00.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.196+0000 7fcb477fe700 1 -- 172.21.15.49:0/1863943033 learned_addr learned my addr 172.21.15.49:0/1863943033 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:00.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.197+0000 7fcb477fe700 1 -- 172.21.15.49:0/1863943033 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb38009550 con 0x7fcb48106a90 2023-07-20T17:42:00.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.197+0000 7fcb477fe700 1 --2- 172.21.15.49:0/1863943033 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb481063f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcb38009fb0 tx=0x7fcb38011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:00.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.197+0000 7fcb44ff9700 1 -- 172.21.15.49:0/1863943033 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb38003a40 con 0x7fcb48106a90 2023-07-20T17:42:00.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.197+0000 7fcb44ff9700 1 -- 172.21.15.49:0/1863943033 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb38003ba0 con 0x7fcb48106a90 2023-07-20T17:42:00.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.197+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1863943033 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb48104a40 con 0x7fcb48106a90 2023-07-20T17:42:00.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.197+0000 7fcb44ff9700 1 -- 172.21.15.49:0/1863943033 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb3801a750 con 0x7fcb48106a90 2023-07-20T17:42:00.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.197+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1863943033 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb48104e60 con 0x7fcb48106a90 2023-07-20T17:42:00.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.197+0000 7fcb44ff9700 1 -- 172.21.15.49:0/1863943033 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcb3801ac80 con 0x7fcb48106a90 2023-07-20T17:42:00.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.197+0000 7fcb44ff9700 1 -- 172.21.15.49:0/1863943033 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcb380193c0 con 0x7fcb48106a90 2023-07-20T17:42:00.761 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.198+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1863943033 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4810a0f0 con 0x7fcb48106a90 2023-07-20T17:42:00.761 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.200+0000 7fcb44ff9700 1 -- 172.21.15.49:0/1863943033 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fcb3802de20 con 0x7fcb48106a90 2023-07-20T17:42:00.761 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.326+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1863943033 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fcb4810a2e0 con 0x7fcb48106a90 2023-07-20T17:42:00.761 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.327+0000 7fcb44ff9700 1 -- 172.21.15.49:0/1863943033 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fcb38024b60 con 0x7fcb48106a90 2023-07-20T17:42:00.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.327+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1863943033 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 msgr2=0x7fcb481063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:00.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.327+0000 7fcb4dffb700 1 --2- 172.21.15.49:0/1863943033 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb481063f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcb38009fb0 tx=0x7fcb38011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:00.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.327+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1863943033 shutdown_connections 2023-07-20T17:42:00.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.327+0000 7fcb4dffb700 1 --2- 172.21.15.49:0/1863943033 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb48106a90 0x7fcb481063f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:00.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.327+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1863943033 >> 172.21.15.49:0/1863943033 conn(0x7fcb48100400 msgr2=0x7fcb48101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:00.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.328+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1863943033 shutdown_connections 2023-07-20T17:42:00.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:00.328+0000 7fcb4dffb700 1 -- 172.21.15.49:0/1863943033 wait complete. 2023-07-20T17:42:00.763 INFO:teuthology.orchestra.run.smithi049.stdout:Restarting the monitor... 2023-07-20T17:42:00.987 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:00 smithi049 systemd[1]: Stopping Ceph mon.a for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:42:01.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:01 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-a[96004]: 2023-07-20T17:42:01.124+0000 7f601080c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:42:01.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:01 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-a[96004]: 2023-07-20T17:42:01.124+0000 7f601080c700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-20T17:42:01.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:01 smithi049 bash[96365]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-a 2023-07-20T17:42:02.098 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:01 smithi049 systemd[1]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.a.service: Succeeded. 2023-07-20T17:42:02.098 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:01 smithi049 systemd[1]: Stopped Ceph mon.a for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:42:02.098 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:01 smithi049 systemd[1]: Starting Ceph mon.a for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:42:02.322 INFO:teuthology.orchestra.run.smithi049.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-20T17:42:02.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 podman[96490]: 2023-07-20T17:42:02.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 bash[96490]: 5e59c597d2ce3c0969583fed075059ff4bb35d3bc86d03a1fca19ecdf4b05050 2023-07-20T17:42:02.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T17:42:02.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T17:42:02.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: pidfile_write: ignore empty --pid-file 2023-07-20T17:42:02.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: load: jerasure load: lrc 2023-07-20T17:42:02.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: RocksDB version: 7.9.2 2023-07-20T17:42:02.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Git sha 0 2023-07-20T17:42:02.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T17:42:02.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: DB SUMMARY 2023-07-20T17:42:02.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: DB Session ID: 2L5N1U7BIYAHWWKERQKP 2023-07-20T17:42:02.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: CURRENT file: CURRENT 2023-07-20T17:42:02.381 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T17:42:02.381 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-20T17:42:02.381 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-20T17:42:02.381 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98677 ; 2023-07-20T17:42:02.381 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.error_if_exists: 0 2023-07-20T17:42:02.381 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.create_if_missing: 0 2023-07-20T17:42:02.381 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.paranoid_checks: 1 2023-07-20T17:42:02.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T17:42:02.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T17:42:02.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T17:42:02.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.env: 0x562c769ebc60 2023-07-20T17:42:02.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T17:42:02.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.info_log: 0x562c77a012c0 2023-07-20T17:42:02.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T17:42:02.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.statistics: (nil) 2023-07-20T17:42:02.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.use_fsync: 0 2023-07-20T17:42:02.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_log_file_size: 0 2023-07-20T17:42:02.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T17:42:02.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T17:42:02.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T17:42:02.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T17:42:02.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.allow_fallocate: 1 2023-07-20T17:42:02.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T17:42:02.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T17:42:02.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.use_direct_reads: 0 2023-07-20T17:42:02.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T17:42:02.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T17:42:02.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.db_log_dir: 2023-07-20T17:42:02.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.wal_dir: 2023-07-20T17:42:02.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T17:42:02.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T17:42:02.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T17:42:02.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T17:42:02.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T17:42:02.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T17:42:02.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T17:42:02.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T17:42:02.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.write_buffer_manager: 0x562c779223c0 2023-07-20T17:42:02.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T17:42:02.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T17:42:02.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T17:42:02.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T17:42:02.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T17:42:02.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T17:42:02.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T17:42:02.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T17:42:02.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.unordered_write: 0 2023-07-20T17:42:02.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T17:42:02.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T17:42:02.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T17:42:02.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T17:42:02.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.row_cache: None 2023-07-20T17:42:02.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.wal_filter: None 2023-07-20T17:42:02.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T17:42:02.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T17:42:02.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.two_write_queues: 0 2023-07-20T17:42:02.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T17:42:02.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.wal_compression: 0 2023-07-20T17:42:02.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.atomic_flush: 0 2023-07-20T17:42:02.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T17:42:02.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T17:42:02.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T17:42:02.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.log_readahead_size: 0 2023-07-20T17:42:02.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T17:42:02.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T17:42:02.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T17:42:02.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T17:42:02.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T17:42:02.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T17:42:02.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T17:42:02.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_background_jobs: 2 2023-07-20T17:42:02.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_background_compactions: -1 2023-07-20T17:42:02.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_subcompactions: 1 2023-07-20T17:42:02.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T17:42:02.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T17:42:02.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T17:42:02.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T17:42:02.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T17:42:02.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T17:42:02.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T17:42:02.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T17:42:02.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_open_files: -1 2023-07-20T17:42:02.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T17:42:02.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T17:42:02.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T17:42:02.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T17:42:02.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_background_flushes: -1 2023-07-20T17:42:02.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Compression algorithms supported: 2023-07-20T17:42:02.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T17:42:02.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: kZSTD supported: 0 2023-07-20T17:42:02.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: kXpressCompression supported: 0 2023-07-20T17:42:02.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T17:42:02.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: kZlibCompression supported: 1 2023-07-20T17:42:02.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: kSnappyCompression supported: 1 2023-07-20T17:42:02.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: kLZ4Compression supported: 1 2023-07-20T17:42:02.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: kBZip2Compression supported: 0 2023-07-20T17:42:02.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T17:42:02.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T17:42:02.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-20T17:42:02.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T17:42:02.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T17:42:02.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.merge_operator: 2023-07-20T17:42:02.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_filter: None 2023-07-20T17:42:02.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T17:42:02.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T17:42:02.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T17:42:02.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T17:42:02.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562c77a013c0) 2023-07-20T17:42:02.398 INFO:journalctl@ceph.mon.a.smithi049.stdout: cache_index_and_filter_blocks: 1 2023-07-20T17:42:02.398 INFO:journalctl@ceph.mon.a.smithi049.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T17:42:02.398 INFO:journalctl@ceph.mon.a.smithi049.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T17:42:02.398 INFO:journalctl@ceph.mon.a.smithi049.stdout: pin_top_level_index_and_filter: 1 2023-07-20T17:42:02.398 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_type: 0 2023-07-20T17:42:02.398 INFO:journalctl@ceph.mon.a.smithi049.stdout: data_block_index_type: 0 2023-07-20T17:42:02.398 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_shortening: 1 2023-07-20T17:42:02.399 INFO:journalctl@ceph.mon.a.smithi049.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T17:42:02.399 INFO:journalctl@ceph.mon.a.smithi049.stdout: checksum: 4 2023-07-20T17:42:02.399 INFO:journalctl@ceph.mon.a.smithi049.stdout: no_block_cache: 0 2023-07-20T17:42:02.399 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache: 0x562c779c1350 2023-07-20T17:42:02.399 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_name: BinnedLRUCache 2023-07-20T17:42:02.399 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_options: 2023-07-20T17:42:02.399 INFO:journalctl@ceph.mon.a.smithi049.stdout: capacity : 536870912 2023-07-20T17:42:02.400 INFO:journalctl@ceph.mon.a.smithi049.stdout: num_shard_bits : 4 2023-07-20T17:42:02.400 INFO:journalctl@ceph.mon.a.smithi049.stdout: strict_capacity_limit : 0 2023-07-20T17:42:02.400 INFO:journalctl@ceph.mon.a.smithi049.stdout: high_pri_pool_ratio: 0.000 2023-07-20T17:42:02.400 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_compressed: (nil) 2023-07-20T17:42:02.400 INFO:journalctl@ceph.mon.a.smithi049.stdout: persistent_cache: (nil) 2023-07-20T17:42:02.400 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_size: 4096 2023-07-20T17:42:02.401 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_size_deviation: 10 2023-07-20T17:42:02.401 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_restart_interval: 16 2023-07-20T17:42:02.401 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_block_restart_interval: 1 2023-07-20T17:42:02.401 INFO:journalctl@ceph.mon.a.smithi049.stdout: metadata_block_size: 4096 2023-07-20T17:42:02.401 INFO:journalctl@ceph.mon.a.smithi049.stdout: partition_filters: 0 2023-07-20T17:42:02.401 INFO:journalctl@ceph.mon.a.smithi049.stdout: use_delta_encoding: 1 2023-07-20T17:42:02.401 INFO:journalctl@ceph.mon.a.smithi049.stdout: filter_policy: bloomfilter 2023-07-20T17:42:02.402 INFO:journalctl@ceph.mon.a.smithi049.stdout: whole_key_filtering: 1 2023-07-20T17:42:02.402 INFO:journalctl@ceph.mon.a.smithi049.stdout: verify_compression: 0 2023-07-20T17:42:02.402 INFO:journalctl@ceph.mon.a.smithi049.stdout: read_amp_bytes_per_bit: 0 2023-07-20T17:42:02.402 INFO:journalctl@ceph.mon.a.smithi049.stdout: format_version: 5 2023-07-20T17:42:02.403 INFO:journalctl@ceph.mon.a.smithi049.stdout: enable_index_compression: 1 2023-07-20T17:42:02.403 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_align: 0 2023-07-20T17:42:02.403 INFO:journalctl@ceph.mon.a.smithi049.stdout: max_auto_readahead_size: 262144 2023-07-20T17:42:02.403 INFO:journalctl@ceph.mon.a.smithi049.stdout: prepopulate_block_cache: 0 2023-07-20T17:42:02.404 INFO:journalctl@ceph.mon.a.smithi049.stdout: initial_auto_readahead_size: 8192 2023-07-20T17:42:02.404 INFO:journalctl@ceph.mon.a.smithi049.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T17:42:02.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T17:42:02.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T17:42:02.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compression: NoCompression 2023-07-20T17:42:02.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T17:42:02.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T17:42:02.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T17:42:02.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.num_levels: 7 2023-07-20T17:42:02.406 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T17:42:02.406 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T17:42:02.406 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T17:42:02.406 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T17:42:02.406 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T17:42:02.406 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T17:42:02.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T17:42:02.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T17:42:02.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T17:42:02.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T17:42:02.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T17:42:02.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T17:42:02.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T17:42:02.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T17:42:02.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T17:42:02.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T17:42:02.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T17:42:02.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T17:42:02.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T17:42:02.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T17:42:02.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T17:42:02.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T17:42:02.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T17:42:02.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T17:42:02.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T17:42:02.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T17:42:02.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T17:42:02.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T17:42:02.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T17:42:02.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T17:42:02.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T17:42:02.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T17:42:02.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T17:42:02.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T17:42:02.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T17:42:02.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T17:42:02.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T17:42:02.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T17:42:02.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T17:42:02.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T17:42:02.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T17:42:02.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T17:42:02.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T17:42:02.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T17:42:02.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T17:42:02.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T17:42:02.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T17:42:02.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T17:42:02.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T17:42:02.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T17:42:02.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T17:42:02.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T17:42:02.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T17:42:02.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.table_properties_collectors: 2023-07-20T17:42:02.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.inplace_update_support: 0 2023-07-20T17:42:02.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T17:42:02.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T17:42:02.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T17:42:02.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T17:42:02.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.bloom_locality: 0 2023-07-20T17:42:02.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.max_successive_merges: 0 2023-07-20T17:42:02.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T17:42:02.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T17:42:02.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T17:42:02.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T17:42:02.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.ttl: 2592000 2023-07-20T17:42:02.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T17:42:02.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T17:42:02.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T17:42:02.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.enable_blob_files: false 2023-07-20T17:42:02.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.min_blob_size: 0 2023-07-20T17:42:02.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T17:42:02.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T17:42:02.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T17:42:02.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T17:42:02.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T17:42:02.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T17:42:02.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T17:42:02.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T17:42:02.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-20T17:42:02.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-20T17:42:02.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e23f7f78-dd38-4f56-94e3-ea09224de41b 2023-07-20T17:42:02.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689874922308300, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-20T17:42:02.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-20T17:42:02.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689874922309274, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93918, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91923, "index_size": 277, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13640, "raw_average_key_size": 49, "raw_value_size": 84488, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689874922, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e23f7f78-dd38-4f56-94e3-ea09224de41b", "db_session_id": "2L5N1U7BIYAHWWKERQKP", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-20T17:42:02.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689874922309352, "job": 1, "event": "recovery_finished"} 2023-07-20T17:42:02.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-20T17:42:02.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T17:42:02.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562c77a5a000 2023-07-20T17:42:02.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: DB pointer 0x562c77a46000 2023-07-20T17:42:02.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T17:42:02.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T17:42:02.419 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** DB Stats ** 2023-07-20T17:42:02.419 INFO:journalctl@ceph.mon.a.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T17:42:02.419 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T17:42:02.419 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T17:42:02.420 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T17:42:02.420 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T17:42:02.420 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T17:42:02.420 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T17:42:02.420 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2023-07-20T17:42:02.424 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** Compaction Stats [default] ** 2023-07-20T17:42:02.424 INFO:journalctl@ceph.mon.a.smithi049.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T17:42:02.425 INFO:journalctl@ceph.mon.a.smithi049.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T17:42:02.425 INFO:journalctl@ceph.mon.a.smithi049.stdout: L0 2/0 93.55 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T17:42:02.425 INFO:journalctl@ceph.mon.a.smithi049.stdout: Sum 2/0 93.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T17:42:02.425 INFO:journalctl@ceph.mon.a.smithi049.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T17:42:02.425 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2023-07-20T17:42:02.425 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** Compaction Stats [default] ** 2023-07-20T17:42:02.426 INFO:journalctl@ceph.mon.a.smithi049.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T17:42:02.426 INFO:journalctl@ceph.mon.a.smithi049.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T17:42:02.426 INFO:journalctl@ceph.mon.a.smithi049.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T17:42:02.426 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2023-07-20T17:42:02.426 INFO:journalctl@ceph.mon.a.smithi049.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T17:42:02.426 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2023-07-20T17:42:02.427 INFO:journalctl@ceph.mon.a.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T17:42:02.427 INFO:journalctl@ceph.mon.a.smithi049.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T17:42:02.427 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T17:42:02.427 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T17:42:02.427 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T17:42:02.427 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T17:42:02.427 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative compaction: 0.00 GB write, 30.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T17:42:02.428 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval compaction: 0.00 GB write, 30.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T17:42:02.428 INFO:journalctl@ceph.mon.a.smithi049.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T17:42:02.428 INFO:journalctl@ceph.mon.a.smithi049.stdout: Block cache BinnedLRUCache@0x562c779c1350#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-20T17:42:02.428 INFO:journalctl@ceph.mon.a.smithi049.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-20T17:42:02.428 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2023-07-20T17:42:02.428 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T17:42:02.428 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: starting mon.a rank 0 at public addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] at bind addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:42:02.429 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: mon.a@-1(???) e1 preinit fsid 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:42:02.429 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: mon.a@-1(???).mds e1 new map 2023-07-20T17:42:02.429 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: mon.a@-1(???).mds e1 print_map 2023-07-20T17:42:02.429 INFO:journalctl@ceph.mon.a.smithi049.stdout: e1 2023-07-20T17:42:02.429 INFO:journalctl@ceph.mon.a.smithi049.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T17:42:02.429 INFO:journalctl@ceph.mon.a.smithi049.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T17:42:02.430 INFO:journalctl@ceph.mon.a.smithi049.stdout: legacy client fscid: -1 2023-07-20T17:42:02.430 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2023-07-20T17:42:02.430 INFO:journalctl@ceph.mon.a.smithi049.stdout: No filesystems configured 2023-07-20T17:42:02.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T17:42:02.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T17:42:02.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T17:42:02.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T17:42:02.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-20T17:42:02.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 systemd[1]: Started Ceph mon.a for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:42:02.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T17:42:02.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: monmap e1: 1 mons at {a=[v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0]} removed_ranks: {} 2023-07-20T17:42:02.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: fsmap 2023-07-20T17:42:02.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T17:42:02.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:02 smithi049 ceph-mon[96530]: mgrmap e1: no daemons active 2023-07-20T17:42:03.473 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.932+0000 7fda54090700 1 Processor -- start 2023-07-20T17:42:03.473 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.932+0000 7fda54090700 1 -- start start 2023-07-20T17:42:03.473 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.932+0000 7fda54090700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c106900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:03.473 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.932+0000 7fda54090700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c106ed0 con 0x7fda4c1064e0 2023-07-20T17:42:03.474 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.933+0000 7fda51e2c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:03.474 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.933+0000 7fda51e2c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35140/0 (socket says 172.21.15.49:35140) 2023-07-20T17:42:03.474 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.933+0000 7fda51e2c700 1 -- 172.21.15.49:0/2395693282 learned_addr learned my addr 172.21.15.49:0/2395693282 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:03.474 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.933+0000 7fda51e2c700 1 -- 172.21.15.49:0/2395693282 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c107010 con 0x7fda4c1064e0 2023-07-20T17:42:03.474 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.934+0000 7fda51e2c700 1 --2- 172.21.15.49:0/2395693282 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c106900 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0098a0 tx=0x7fda3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e73627e58c3a224 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:03.474 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.934+0000 7fda50e2a700 1 -- 172.21.15.49:0/2395693282 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda3c004160 con 0x7fda4c1064e0 2023-07-20T17:42:03.474 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.934+0000 7fda50e2a700 1 -- 172.21.15.49:0/2395693282 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda3c0042c0 con 0x7fda4c1064e0 2023-07-20T17:42:03.475 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.934+0000 7fda50e2a700 1 -- 172.21.15.49:0/2395693282 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda3c011a00 con 0x7fda4c1064e0 2023-07-20T17:42:03.475 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.934+0000 7fda54090700 1 -- 172.21.15.49:0/2395693282 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 msgr2=0x7fda4c106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:03.475 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.934+0000 7fda54090700 1 --2- 172.21.15.49:0/2395693282 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c106900 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0098a0 tx=0x7fda3c009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:03.475 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.935+0000 7fda54090700 1 -- 172.21.15.49:0/2395693282 shutdown_connections 2023-07-20T17:42:03.475 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.935+0000 7fda54090700 1 --2- 172.21.15.49:0/2395693282 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c106900 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:03.475 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.935+0000 7fda54090700 1 -- 172.21.15.49:0/2395693282 >> 172.21.15.49:0/2395693282 conn(0x7fda4c0ffbc0 msgr2=0x7fda4c101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:03.475 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.935+0000 7fda54090700 1 -- 172.21.15.49:0/2395693282 shutdown_connections 2023-07-20T17:42:03.475 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.935+0000 7fda54090700 1 -- 172.21.15.49:0/2395693282 wait complete. 2023-07-20T17:42:03.476 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.935+0000 7fda54090700 1 Processor -- start 2023-07-20T17:42:03.476 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.935+0000 7fda54090700 1 -- start start 2023-07-20T17:42:03.476 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda54090700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c19be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:03.476 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda54090700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c19c350 con 0x7fda4c1064e0 2023-07-20T17:42:03.476 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda51e2c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c19be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:03.476 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda51e2c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c19be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35142/0 (socket says 172.21.15.49:35142) 2023-07-20T17:42:03.477 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda51e2c700 1 -- 172.21.15.49:0/4207501652 learned_addr learned my addr 172.21.15.49:0/4207501652 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:03.477 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda51e2c700 1 -- 172.21.15.49:0/4207501652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c009550 con 0x7fda4c1064e0 2023-07-20T17:42:03.477 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda51e2c700 1 --2- 172.21.15.49:0/4207501652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c19be10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009130 tx=0x7fda3c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:03.477 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda42ffd700 1 -- 172.21.15.49:0/4207501652 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda3c003b70 con 0x7fda4c1064e0 2023-07-20T17:42:03.477 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda42ffd700 1 -- 172.21.15.49:0/4207501652 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda3c003cd0 con 0x7fda4c1064e0 2023-07-20T17:42:03.477 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda54090700 1 -- 172.21.15.49:0/4207501652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4c19c550 con 0x7fda4c1064e0 2023-07-20T17:42:03.477 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda42ffd700 1 -- 172.21.15.49:0/4207501652 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda3c0197d0 con 0x7fda4c1064e0 2023-07-20T17:42:03.477 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.936+0000 7fda54090700 1 -- 172.21.15.49:0/4207501652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4c19c970 con 0x7fda4c1064e0 2023-07-20T17:42:03.478 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.937+0000 7fda42ffd700 1 -- 172.21.15.49:0/4207501652 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fda3c019a10 con 0x7fda4c1064e0 2023-07-20T17:42:03.478 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.937+0000 7fda42ffd700 1 -- 172.21.15.49:0/4207501652 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fda3c00b720 con 0x7fda4c1064e0 2023-07-20T17:42:03.478 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.938+0000 7fda54090700 1 -- 172.21.15.49:0/4207501652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4c195b40 con 0x7fda4c1064e0 2023-07-20T17:42:03.478 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:02.940+0000 7fda42ffd700 1 -- 172.21.15.49:0/4207501652 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fda3c02a060 con 0x7fda4c1064e0 2023-07-20T17:42:03.478 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:03.065+0000 7fda54090700 1 -- 172.21.15.49:0/4207501652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fda4c061da0 con 0x7fda4c1064e0 2023-07-20T17:42:03.478 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:03.068+0000 7fda42ffd700 1 -- 172.21.15.49:0/4207501652 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fda3c019e50 con 0x7fda4c1064e0 2023-07-20T17:42:03.478 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:03.068+0000 7fda54090700 1 -- 172.21.15.49:0/4207501652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 msgr2=0x7fda4c19be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:03.479 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:03.069+0000 7fda54090700 1 --2- 172.21.15.49:0/4207501652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c19be10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009130 tx=0x7fda3c011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:03.479 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:03.069+0000 7fda54090700 1 -- 172.21.15.49:0/4207501652 shutdown_connections 2023-07-20T17:42:03.479 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:03.069+0000 7fda54090700 1 --2- 172.21.15.49:0/4207501652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda4c1064e0 0x7fda4c19be10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:03.479 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:03.069+0000 7fda54090700 1 -- 172.21.15.49:0/4207501652 >> 172.21.15.49:0/4207501652 conn(0x7fda4c0ffbc0 msgr2=0x7fda4c192c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:03.479 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:03.069+0000 7fda54090700 1 -- 172.21.15.49:0/4207501652 shutdown_connections 2023-07-20T17:42:03.479 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:03.069+0000 7fda54090700 1 -- 172.21.15.49:0/4207501652 wait complete. 2023-07-20T17:42:03.480 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-20T17:42:03.494 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-20T17:42:03.494 INFO:teuthology.orchestra.run.smithi049.stdout:Creating mgr... 2023-07-20T17:42:03.495 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying port 9283 ... 2023-07-20T17:42:03.495 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying port 8765 ... 2023-07-20T17:42:03.672 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T17:42:03.673 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.x.service: Unit ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.x.service not loaded. 2023-07-20T17:42:03.766 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8c050cea-2724-11ee-9b34-001a4aab830c.target.wants/ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-8c050cea-2724-11ee-9b34-001a4aab830c@.service. 2023-07-20T17:42:03.987 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:03 smithi049 systemd[1]: Starting Ceph mgr.x for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:42:04.358 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:04 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/4207501652' entity='client.admin' 2023-07-20T17:42:04.359 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:04 smithi049 podman[96761]: 2023-07-20T17:42:04.475 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T17:42:04.475 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2023-07-20T17:42:04.484 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T17:42:04.484 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2023-07-20T17:42:04.485 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2023-07-20T17:42:04.485 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T17:42:04.494 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T17:42:04.495 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2023-07-20T17:42:04.502 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T17:42:04.502 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2023-07-20T17:42:04.502 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2023-07-20T17:42:04.503 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-20T17:42:04.503 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr to start... 2023-07-20T17:42:04.503 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr... 2023-07-20T17:42:04.714 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:04 smithi049 bash[96761]: 9ea35b4aee0f280927885a4d21587ed1a77db8374806aedd4b59fd6aabc2c4f8 2023-07-20T17:42:04.714 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:04 smithi049 systemd[1]: Started Ceph mgr.x for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:42:04.714 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:04.503+0000 7ff76b604200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T17:42:04.714 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:04.602+0000 7ff76b604200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T17:42:04.988 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:04.714+0000 7ff76b604200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T17:42:04.988 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:04.829+0000 7ff76b604200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T17:42:05.326 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:05 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3193382661' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T17:42:05.327 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:05 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:05.043+0000 7ff76b604200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T17:42:05.554 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:42:05.555 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2023-07-20T17:42:05.555 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "8c050cea-2724-11ee-9b34-001a4aab830c", 2023-07-20T17:42:05.555 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T17:42:05.555 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T17:42:05.555 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T17:42:05.555 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T17:42:05.556 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:05.556 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T17:42:05.556 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T17:42:05.556 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2023-07-20T17:42:05.556 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:05.556 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T17:42:05.556 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T17:42:05.557 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:05.557 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-07-20T17:42:05.557 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T17:42:05.557 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:05.557 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T17:42:05.557 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T17:42:05.557 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:05.558 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T17:42:05.558 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:05.558 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T17:42:05.558 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T17:42:05.558 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T17:42:05.565 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T17:42:05.565 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T17:42:05.565 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T17:42:05.565 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:05.565 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T17:42:05.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T17:42:05.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T17:42:05.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T17:42:05.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T17:42:05.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T17:42:05.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T17:42:05.566 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T17:42:05.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T17:42:05.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:05.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T17:42:05.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:05.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T17:42:05.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T17:42:05.567 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:05.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T17:42:05.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T17:42:05.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T17:42:05.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T17:42:05.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T17:42:05.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T17:42:05.568 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T17:42:05.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:05.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T17:42:05.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:05.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T17:42:05.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:05.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T17:41:57.467787+0000", 2023-07-20T17:42:05.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T17:42:05.569 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:05.570 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T17:42:05.570 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2023-07-20T17:42:05.570 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.029+0000 7fb7a0bf5700 1 Processor -- start 2023-07-20T17:42:05.570 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.029+0000 7fb7a0bf5700 1 -- start start 2023-07-20T17:42:05.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.030+0000 7fb7a0bf5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c108e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:05.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.030+0000 7fb7a0bf5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c079e70 con 0x7fb79c106a10 2023-07-20T17:42:05.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.030+0000 7fb79a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c108e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:05.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.030+0000 7fb79a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c108e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35148/0 (socket says 172.21.15.49:35148) 2023-07-20T17:42:05.571 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.030+0000 7fb79a59c700 1 -- 172.21.15.49:0/3158196828 learned_addr learned my addr 172.21.15.49:0/3158196828 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:05.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.030+0000 7fb79a59c700 1 -- 172.21.15.49:0/3158196828 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c109340 con 0x7fb79c106a10 2023-07-20T17:42:05.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.031+0000 7fb79a59c700 1 --2- 172.21.15.49:0/3158196828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c108e00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb78c0098a0 tx=0x7fb78c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb72022c264c6e server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:05.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.031+0000 7fb79959a700 1 -- 172.21.15.49:0/3158196828 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb78c004160 con 0x7fb79c106a10 2023-07-20T17:42:05.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.031+0000 7fb79959a700 1 -- 172.21.15.49:0/3158196828 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb78c0042c0 con 0x7fb79c106a10 2023-07-20T17:42:05.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.031+0000 7fb79959a700 1 -- 172.21.15.49:0/3158196828 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb78c011a00 con 0x7fb79c106a10 2023-07-20T17:42:05.572 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.032+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3158196828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 msgr2=0x7fb79c108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:05.573 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.032+0000 7fb7a0bf5700 1 --2- 172.21.15.49:0/3158196828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c108e00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb78c0098a0 tx=0x7fb78c009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:05.573 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.032+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3158196828 shutdown_connections 2023-07-20T17:42:05.573 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.032+0000 7fb7a0bf5700 1 --2- 172.21.15.49:0/3158196828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c108e00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:05.573 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.032+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3158196828 >> 172.21.15.49:0/3158196828 conn(0x7fb79c100400 msgr2=0x7fb79c102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:05.574 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.032+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3158196828 shutdown_connections 2023-07-20T17:42:05.574 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.032+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3158196828 wait complete. 2023-07-20T17:42:05.574 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.033+0000 7fb7a0bf5700 1 Processor -- start 2023-07-20T17:42:05.574 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.033+0000 7fb7a0bf5700 1 -- start start 2023-07-20T17:42:05.575 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.033+0000 7fb7a0bf5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:05.575 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.033+0000 7fb7a0bf5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c19a180 con 0x7fb79c106a10 2023-07-20T17:42:05.575 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.034+0000 7fb79a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:05.576 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.034+0000 7fb79a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35150/0 (socket says 172.21.15.49:35150) 2023-07-20T17:42:05.576 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.034+0000 7fb79a59c700 1 -- 172.21.15.49:0/3193382661 learned_addr learned my addr 172.21.15.49:0/3193382661 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:05.577 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.034+0000 7fb79a59c700 1 -- 172.21.15.49:0/3193382661 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c009550 con 0x7fb79c106a10 2023-07-20T17:42:05.577 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.034+0000 7fb79a59c700 1 --2- 172.21.15.49:0/3193382661 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c199c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009fb0 tx=0x7fb78c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:05.577 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.034+0000 7fb78b7fe700 1 -- 172.21.15.49:0/3193382661 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb78c003a40 con 0x7fb79c106a10 2023-07-20T17:42:05.578 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.034+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3193382661 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb79c19a380 con 0x7fb79c106a10 2023-07-20T17:42:05.578 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.034+0000 7fb78b7fe700 1 -- 172.21.15.49:0/3193382661 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb78c003ba0 con 0x7fb79c106a10 2023-07-20T17:42:05.578 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.034+0000 7fb78b7fe700 1 -- 172.21.15.49:0/3193382661 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb78c01a750 con 0x7fb79c106a10 2023-07-20T17:42:05.578 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.034+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3193382661 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb79c19a7a0 con 0x7fb79c106a10 2023-07-20T17:42:05.579 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.035+0000 7fb78b7fe700 1 -- 172.21.15.49:0/3193382661 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb78c01acd0 con 0x7fb79c106a10 2023-07-20T17:42:05.579 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.035+0000 7fb78b7fe700 1 -- 172.21.15.49:0/3193382661 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb78c0193c0 con 0x7fb79c106a10 2023-07-20T17:42:05.579 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.035+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3193382661 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79c061da0 con 0x7fb79c106a10 2023-07-20T17:42:05.579 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.038+0000 7fb78b7fe700 1 -- 172.21.15.49:0/3193382661 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb78c02de40 con 0x7fb79c106a10 2023-07-20T17:42:05.580 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.170+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3193382661 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb79c19d290 con 0x7fb79c106a10 2023-07-20T17:42:05.580 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.170+0000 7fb78b7fe700 1 -- 172.21.15.49:0/3193382661 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fb78c024c10 con 0x7fb79c106a10 2023-07-20T17:42:05.580 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.171+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3193382661 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 msgr2=0x7fb79c199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:05.581 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.171+0000 7fb7a0bf5700 1 --2- 172.21.15.49:0/3193382661 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c199c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009fb0 tx=0x7fb78c011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:05.581 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.171+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3193382661 shutdown_connections 2023-07-20T17:42:05.581 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.171+0000 7fb7a0bf5700 1 --2- 172.21.15.49:0/3193382661 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb79c106a10 0x7fb79c199c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:05.581 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.171+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3193382661 >> 172.21.15.49:0/3193382661 conn(0x7fb79c100400 msgr2=0x7fb79c101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:05.582 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.172+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3193382661 shutdown_connections 2023-07-20T17:42:05.582 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:05.172+0000 7fb7a0bf5700 1 -- 172.21.15.49:0/3193382661 wait complete. 2023-07-20T17:42:05.582 INFO:teuthology.orchestra.run.smithi049.stdout:mgr not available, waiting (1/15)... 2023-07-20T17:42:05.715 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:05 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:05.452+0000 7ff76b604200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T17:42:05.715 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:05 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:05.564+0000 7ff76b604200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T17:42:05.716 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:05 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:05.715+0000 7ff76b604200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T17:42:05.987 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:05 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:05.830+0000 7ff76b604200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T17:42:06.737 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:06 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:06.339+0000 7ff76b604200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T17:42:07.487 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:07 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:07.033+0000 7ff76b604200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T17:42:07.487 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:07 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:07.224+0000 7ff76b604200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T17:42:08.487 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:08 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2993900081' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T17:42:08.488 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:08 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:08.200+0000 7ff76b604200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T17:42:08.488 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:08 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:08.318+0000 7ff76b604200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T17:42:08.612 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:42:08.612 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2023-07-20T17:42:08.612 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "8c050cea-2724-11ee-9b34-001a4aab830c", 2023-07-20T17:42:08.612 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T17:42:08.613 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T17:42:08.613 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T17:42:08.613 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T17:42:08.613 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:08.613 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T17:42:08.613 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T17:42:08.613 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2023-07-20T17:42:08.613 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:08.614 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T17:42:08.614 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T17:42:08.614 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:08.614 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-07-20T17:42:08.614 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T17:42:08.614 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:08.614 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T17:42:08.614 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T17:42:08.615 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:08.615 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T17:42:08.615 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:08.615 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T17:42:08.615 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T17:42:08.615 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T17:42:08.615 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T17:42:08.615 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T17:42:08.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T17:42:08.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:08.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T17:42:08.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T17:42:08.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T17:42:08.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T17:42:08.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T17:42:08.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T17:42:08.617 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T17:42:08.617 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T17:42:08.617 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T17:42:08.617 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:08.617 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T17:42:08.617 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:08.617 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T17:42:08.617 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T17:42:08.618 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:08.618 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T17:42:08.618 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T17:42:08.618 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T17:42:08.618 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T17:42:08.618 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T17:42:08.618 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T17:42:08.618 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T17:42:08.619 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:08.619 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T17:42:08.619 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:08.619 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T17:42:08.619 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:08.619 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T17:41:57.467787+0000", 2023-07-20T17:42:08.619 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T17:42:08.620 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:08.620 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T17:42:08.620 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2023-07-20T17:42:08.620 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.074+0000 7f19bf8dc700 1 Processor -- start 2023-07-20T17:42:08.620 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.074+0000 7f19bf8dc700 1 -- start start 2023-07-20T17:42:08.620 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.074+0000 7f19bf8dc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b8105280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:08.620 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.074+0000 7f19bf8dc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b81057c0 con 0x7f19b8104e60 2023-07-20T17:42:08.620 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.075+0000 7f19bd678700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b8105280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:08.621 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.075+0000 7f19bd678700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b8105280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35152/0 (socket says 172.21.15.49:35152) 2023-07-20T17:42:08.621 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.075+0000 7f19bd678700 1 -- 172.21.15.49:0/100118276 learned_addr learned my addr 172.21.15.49:0/100118276 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:08.621 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.075+0000 7f19bd678700 1 -- 172.21.15.49:0/100118276 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b8105900 con 0x7f19b8104e60 2023-07-20T17:42:08.621 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.075+0000 7f19bd678700 1 --2- 172.21.15.49:0/100118276 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b8105280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f19b400bb40 tx=0x7f19b400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a7d6774f443278 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:08.621 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.075+0000 7f19affff700 1 -- 172.21.15.49:0/100118276 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19b400dbe0 con 0x7f19b8104e60 2023-07-20T17:42:08.621 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.075+0000 7f19affff700 1 -- 172.21.15.49:0/100118276 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19b40044d0 con 0x7f19b8104e60 2023-07-20T17:42:08.621 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.075+0000 7f19affff700 1 -- 172.21.15.49:0/100118276 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19b4010470 con 0x7f19b8104e60 2023-07-20T17:42:08.621 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.076+0000 7f19bf8dc700 1 -- 172.21.15.49:0/100118276 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 msgr2=0x7f19b8105280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:08.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.076+0000 7f19bf8dc700 1 --2- 172.21.15.49:0/100118276 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b8105280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f19b400bb40 tx=0x7f19b400be50 comp rx=0 tx=0).stop 2023-07-20T17:42:08.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.076+0000 7f19bf8dc700 1 -- 172.21.15.49:0/100118276 shutdown_connections 2023-07-20T17:42:08.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.076+0000 7f19bf8dc700 1 --2- 172.21.15.49:0/100118276 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b8105280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:08.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.076+0000 7f19bf8dc700 1 -- 172.21.15.49:0/100118276 >> 172.21.15.49:0/100118276 conn(0x7f19b8100400 msgr2=0x7f19b8102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:08.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.076+0000 7f19bf8dc700 1 -- 172.21.15.49:0/100118276 shutdown_connections 2023-07-20T17:42:08.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.076+0000 7f19bf8dc700 1 -- 172.21.15.49:0/100118276 wait complete. 2023-07-20T17:42:08.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.077+0000 7f19bf8dc700 1 Processor -- start 2023-07-20T17:42:08.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.077+0000 7f19bf8dc700 1 -- start start 2023-07-20T17:42:08.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.077+0000 7f19bf8dc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b81979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:08.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.077+0000 7f19bf8dc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b8197f30 con 0x7f19b8104e60 2023-07-20T17:42:08.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.077+0000 7f19bd678700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b81979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:08.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.077+0000 7f19bd678700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b81979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35154/0 (socket says 172.21.15.49:35154) 2023-07-20T17:42:08.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.077+0000 7f19bd678700 1 -- 172.21.15.49:0/2993900081 learned_addr learned my addr 172.21.15.49:0/2993900081 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:08.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.078+0000 7f19bd678700 1 -- 172.21.15.49:0/2993900081 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b400b7f0 con 0x7f19b8104e60 2023-07-20T17:42:08.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.078+0000 7f19bd678700 1 --2- 172.21.15.49:0/2993900081 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b81979f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f19b4003b20 tx=0x7f19b4003980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:08.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.078+0000 7f19ae7fc700 1 -- 172.21.15.49:0/2993900081 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19b401e070 con 0x7f19b8104e60 2023-07-20T17:42:08.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.078+0000 7f19ae7fc700 1 -- 172.21.15.49:0/2993900081 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19b4004300 con 0x7f19b8104e60 2023-07-20T17:42:08.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.078+0000 7f19bf8dc700 1 -- 172.21.15.49:0/2993900081 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b8198130 con 0x7f19b8104e60 2023-07-20T17:42:08.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.078+0000 7f19ae7fc700 1 -- 172.21.15.49:0/2993900081 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19b4022a50 con 0x7f19b8104e60 2023-07-20T17:42:08.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.078+0000 7f19bf8dc700 1 -- 172.21.15.49:0/2993900081 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b8198550 con 0x7f19b8104e60 2023-07-20T17:42:08.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.079+0000 7f19ae7fc700 1 -- 172.21.15.49:0/2993900081 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f19b402c740 con 0x7f19b8104e60 2023-07-20T17:42:08.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.079+0000 7f19ae7fc700 1 -- 172.21.15.49:0/2993900081 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f19b4019920 con 0x7f19b8104e60 2023-07-20T17:42:08.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.079+0000 7f19bf8dc700 1 -- 172.21.15.49:0/2993900081 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b81916e0 con 0x7f19b8104e60 2023-07-20T17:42:08.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.081+0000 7f19ae7fc700 1 -- 172.21.15.49:0/2993900081 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f19b4027070 con 0x7f19b8104e60 2023-07-20T17:42:08.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.214+0000 7f19bf8dc700 1 -- 172.21.15.49:0/2993900081 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f19b8061da0 con 0x7f19b8104e60 2023-07-20T17:42:08.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.214+0000 7f19ae7fc700 1 -- 172.21.15.49:0/2993900081 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f19b4010e40 con 0x7f19b8104e60 2023-07-20T17:42:08.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.215+0000 7f19bf8dc700 1 -- 172.21.15.49:0/2993900081 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 msgr2=0x7f19b81979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:08.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.215+0000 7f19bf8dc700 1 --2- 172.21.15.49:0/2993900081 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b81979f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f19b4003b20 tx=0x7f19b4003980 comp rx=0 tx=0).stop 2023-07-20T17:42:08.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.215+0000 7f19bf8dc700 1 -- 172.21.15.49:0/2993900081 shutdown_connections 2023-07-20T17:42:08.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.215+0000 7f19bf8dc700 1 --2- 172.21.15.49:0/2993900081 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19b8104e60 0x7f19b81979f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:08.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.215+0000 7f19bf8dc700 1 -- 172.21.15.49:0/2993900081 >> 172.21.15.49:0/2993900081 conn(0x7f19b8100400 msgr2=0x7f19b818e820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:08.626 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.215+0000 7f19bf8dc700 1 -- 172.21.15.49:0/2993900081 shutdown_connections 2023-07-20T17:42:08.626 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:08.215+0000 7f19bf8dc700 1 -- 172.21.15.49:0/2993900081 wait complete. 2023-07-20T17:42:08.626 INFO:teuthology.orchestra.run.smithi049.stdout:mgr not available, waiting (2/15)... 2023-07-20T17:42:08.885 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:08 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:08.537+0000 7ff76b604200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T17:42:09.138 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:08 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:08.885+0000 7ff76b604200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T17:42:09.487 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:09 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:09.138+0000 7ff76b604200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T17:42:09.487 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:09 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:09.319+0000 7ff76b604200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T17:42:09.987 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:09 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:09.644+0000 7ff76b604200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T17:42:10.737 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:10 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:10.463+0000 7ff76b604200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T17:42:10.738 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:10 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:10.619+0000 7ff76b604200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T17:42:11.237 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:10 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:10.868+0000 7ff76b604200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T17:42:11.238 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:11 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:11.023+0000 7ff76b604200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T17:42:11.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:11 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2654986125' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T17:42:11.698 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:11 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:11.311+0000 7ff76b604200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T17:42:11.756 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:42:11.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2023-07-20T17:42:11.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "8c050cea-2724-11ee-9b34-001a4aab830c", 2023-07-20T17:42:11.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T17:42:11.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T17:42:11.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T17:42:11.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T17:42:11.757 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:11.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T17:42:11.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T17:42:11.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2023-07-20T17:42:11.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:11.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T17:42:11.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T17:42:11.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:11.758 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-07-20T17:42:11.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T17:42:11.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:11.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T17:42:11.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T17:42:11.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:11.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T17:42:11.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:11.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T17:42:11.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T17:42:11.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T17:42:11.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T17:42:11.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T17:42:11.766 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T17:42:11.766 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:11.766 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T17:42:11.766 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T17:42:11.766 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T17:42:11.766 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T17:42:11.766 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T17:42:11.766 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T17:42:11.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T17:42:11.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T17:42:11.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T17:42:11.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:11.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T17:42:11.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:11.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T17:42:11.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T17:42:11.768 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:11.768 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T17:42:11.768 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T17:42:11.768 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T17:42:11.768 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T17:42:11.768 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T17:42:11.768 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T17:42:11.769 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T17:42:11.769 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:11.769 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T17:42:11.769 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:11.769 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T17:42:11.769 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:11.769 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T17:41:57.467787+0000", 2023-07-20T17:42:11.769 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T17:42:11.770 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:11.770 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T17:42:11.770 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2023-07-20T17:42:11.770 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.169+0000 7fb22b273700 1 Processor -- start 2023-07-20T17:42:11.770 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.170+0000 7fb22b273700 1 -- start start 2023-07-20T17:42:11.770 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.170+0000 7fb22b273700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224108dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:11.770 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.170+0000 7fb22b273700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb224079eb0 con 0x7fb2241069d0 2023-07-20T17:42:11.770 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.170+0000 7fb22900f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224108dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:11.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.170+0000 7fb22900f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224108dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35156/0 (socket says 172.21.15.49:35156) 2023-07-20T17:42:11.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.170+0000 7fb22900f700 1 -- 172.21.15.49:0/3309324719 learned_addr learned my addr 172.21.15.49:0/3309324719 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:11.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.170+0000 7fb22900f700 1 -- 172.21.15.49:0/3309324719 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb224109300 con 0x7fb2241069d0 2023-07-20T17:42:11.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.171+0000 7fb22900f700 1 --2- 172.21.15.49:0/3309324719 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224108dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb2200098a0 tx=0x7fb220009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862ff348d35cc1a4 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:11.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.171+0000 7fb21bfff700 1 -- 172.21.15.49:0/3309324719 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb220004160 con 0x7fb2241069d0 2023-07-20T17:42:11.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.171+0000 7fb21bfff700 1 -- 172.21.15.49:0/3309324719 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2200042c0 con 0x7fb2241069d0 2023-07-20T17:42:11.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.171+0000 7fb21bfff700 1 -- 172.21.15.49:0/3309324719 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb220011a00 con 0x7fb2241069d0 2023-07-20T17:42:11.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.171+0000 7fb22b273700 1 -- 172.21.15.49:0/3309324719 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 msgr2=0x7fb224108dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:11.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.171+0000 7fb22b273700 1 --2- 172.21.15.49:0/3309324719 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224108dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb2200098a0 tx=0x7fb220009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:11.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.171+0000 7fb22b273700 1 -- 172.21.15.49:0/3309324719 shutdown_connections 2023-07-20T17:42:11.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.171+0000 7fb22b273700 1 --2- 172.21.15.49:0/3309324719 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224108dc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:11.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.172+0000 7fb22b273700 1 -- 172.21.15.49:0/3309324719 >> 172.21.15.49:0/3309324719 conn(0x7fb224100400 msgr2=0x7fb224102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:11.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.172+0000 7fb22b273700 1 -- 172.21.15.49:0/3309324719 shutdown_connections 2023-07-20T17:42:11.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.172+0000 7fb22b273700 1 -- 172.21.15.49:0/3309324719 wait complete. 2023-07-20T17:42:11.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.172+0000 7fb22b273700 1 Processor -- start 2023-07-20T17:42:11.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.172+0000 7fb22b273700 1 -- start start 2023-07-20T17:42:11.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.172+0000 7fb22b273700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224106330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:11.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.172+0000 7fb22b273700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb224106870 con 0x7fb2241069d0 2023-07-20T17:42:11.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.173+0000 7fb22900f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224106330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:11.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.173+0000 7fb22900f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224106330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35158/0 (socket says 172.21.15.49:35158) 2023-07-20T17:42:11.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.173+0000 7fb22900f700 1 -- 172.21.15.49:0/2654986125 learned_addr learned my addr 172.21.15.49:0/2654986125 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:11.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.173+0000 7fb22900f700 1 -- 172.21.15.49:0/2654986125 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb220009550 con 0x7fb2241069d0 2023-07-20T17:42:11.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.173+0000 7fb22900f700 1 --2- 172.21.15.49:0/2654986125 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224106330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb220009130 tx=0x7fb220011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:11.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.173+0000 7fb21a7fc700 1 -- 172.21.15.49:0/2654986125 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb220003b70 con 0x7fb2241069d0 2023-07-20T17:42:11.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.173+0000 7fb21a7fc700 1 -- 172.21.15.49:0/2654986125 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb220003cd0 con 0x7fb2241069d0 2023-07-20T17:42:11.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.173+0000 7fb22b273700 1 -- 172.21.15.49:0/2654986125 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb224104980 con 0x7fb2241069d0 2023-07-20T17:42:11.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.173+0000 7fb21a7fc700 1 -- 172.21.15.49:0/2654986125 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2200197d0 con 0x7fb2241069d0 2023-07-20T17:42:11.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.173+0000 7fb22b273700 1 -- 172.21.15.49:0/2654986125 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb224104da0 con 0x7fb2241069d0 2023-07-20T17:42:11.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.174+0000 7fb21a7fc700 1 -- 172.21.15.49:0/2654986125 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb2200199c0 con 0x7fb2241069d0 2023-07-20T17:42:11.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.174+0000 7fb21a7fc700 1 -- 172.21.15.49:0/2654986125 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb22000b720 con 0x7fb2241069d0 2023-07-20T17:42:11.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.174+0000 7fb22b273700 1 -- 172.21.15.49:0/2654986125 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb224061da0 con 0x7fb2241069d0 2023-07-20T17:42:11.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.176+0000 7fb21a7fc700 1 -- 172.21.15.49:0/2654986125 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb22002a060 con 0x7fb2241069d0 2023-07-20T17:42:11.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.321+0000 7fb22b273700 1 -- 172.21.15.49:0/2654986125 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb2241050a0 con 0x7fb2241069d0 2023-07-20T17:42:11.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.321+0000 7fb21a7fc700 1 -- 172.21.15.49:0/2654986125 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fb220019e00 con 0x7fb2241069d0 2023-07-20T17:42:11.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.322+0000 7fb22b273700 1 -- 172.21.15.49:0/2654986125 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 msgr2=0x7fb224106330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:11.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.322+0000 7fb22b273700 1 --2- 172.21.15.49:0/2654986125 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224106330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb220009130 tx=0x7fb220011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:11.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.322+0000 7fb22b273700 1 -- 172.21.15.49:0/2654986125 shutdown_connections 2023-07-20T17:42:11.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.322+0000 7fb22b273700 1 --2- 172.21.15.49:0/2654986125 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2241069d0 0x7fb224106330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:11.776 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.322+0000 7fb22b273700 1 -- 172.21.15.49:0/2654986125 >> 172.21.15.49:0/2654986125 conn(0x7fb224100400 msgr2=0x7fb224101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:11.776 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.322+0000 7fb22b273700 1 -- 172.21.15.49:0/2654986125 shutdown_connections 2023-07-20T17:42:11.776 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:11.322+0000 7fb22b273700 1 -- 172.21.15.49:0/2654986125 wait complete. 2023-07-20T17:42:11.776 INFO:teuthology.orchestra.run.smithi049.stdout:mgr not available, waiting (3/15)... 2023-07-20T17:42:12.737 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:12 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:12.360+0000 7ff76b604200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T17:42:13.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: Activating manager daemon x 2023-07-20T17:42:13.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: mgrmap e2: x(active, starting, since 0.00294837s) 2023-07-20T17:42:13.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: from='mgr.14100 172.21.15.49:0/1972453709' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T17:42:13.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: from='mgr.14100 172.21.15.49:0/1972453709' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T17:42:13.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: from='mgr.14100 172.21.15.49:0/1972453709' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T17:42:13.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: from='mgr.14100 172.21.15.49:0/1972453709' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:42:13.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: from='mgr.14100 172.21.15.49:0/1972453709' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T17:42:13.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: Manager daemon x is now available 2023-07-20T17:42:13.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: from='mgr.14100 172.21.15.49:0/1972453709' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T17:42:13.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: from='mgr.14100 172.21.15.49:0/1972453709' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T17:42:13.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: from='mgr.14100 172.21.15.49:0/1972453709' entity='mgr.x' 2023-07-20T17:42:13.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: from='mgr.14100 172.21.15.49:0/1972453709' entity='mgr.x' 2023-07-20T17:42:13.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:13 smithi049 ceph-mon[96530]: from='mgr.14100 172.21.15.49:0/1972453709' entity='mgr.x' 2023-07-20T17:42:14.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:14 smithi049 ceph-mon[96530]: mgrmap e3: x(active, since 1.00726s) 2023-07-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2023-07-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "8c050cea-2724-11ee-9b34-001a4aab830c", 2023-07-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2023-07-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2023-07-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2023-07-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T17:41:57.467787+0000", 2023-07-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2023-07-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2023-07-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.280+0000 7f287435b700 1 Processor -- start 2023-07-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.280+0000 7f287435b700 1 -- start start 2023-07-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.280+0000 7f287435b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c108ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.280+0000 7f287435b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c079dd0 con 0x7f286c106ab0 2023-07-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.280+0000 7f28720f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c108ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.280+0000 7f28720f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c108ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35184/0 (socket says 172.21.15.49:35184) 2023-07-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.280+0000 7f28720f7700 1 -- 172.21.15.49:0/3953134662 learned_addr learned my addr 172.21.15.49:0/3953134662 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.281+0000 7f28720f7700 1 -- 172.21.15.49:0/3953134662 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c079f10 con 0x7f286c106ab0 2023-07-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.281+0000 7f28720f7700 1 --2- 172.21.15.49:0/3953134662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c108ea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f28680098a0 tx=0x7f2868009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434613e44e70b782 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.281+0000 7f28710f5700 1 -- 172.21.15.49:0/3953134662 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2868004160 con 0x7f286c106ab0 2023-07-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.281+0000 7f28710f5700 1 -- 172.21.15.49:0/3953134662 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28680042c0 con 0x7f286c106ab0 2023-07-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.281+0000 7f28710f5700 1 -- 172.21.15.49:0/3953134662 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2868011a00 con 0x7f286c106ab0 2023-07-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.281+0000 7f287435b700 1 -- 172.21.15.49:0/3953134662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 msgr2=0x7f286c108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.281+0000 7f287435b700 1 --2- 172.21.15.49:0/3953134662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c108ea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f28680098a0 tx=0x7f2868009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.282+0000 7f287435b700 1 -- 172.21.15.49:0/3953134662 shutdown_connections 2023-07-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.282+0000 7f287435b700 1 --2- 172.21.15.49:0/3953134662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c108ea0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.282+0000 7f287435b700 1 -- 172.21.15.49:0/3953134662 >> 172.21.15.49:0/3953134662 conn(0x7f286c100400 msgr2=0x7f286c102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.282+0000 7f287435b700 1 -- 172.21.15.49:0/3953134662 shutdown_connections 2023-07-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.282+0000 7f287435b700 1 -- 172.21.15.49:0/3953134662 wait complete. 2023-07-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.282+0000 7f287435b700 1 Processor -- start 2023-07-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.282+0000 7f287435b700 1 -- start start 2023-07-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.282+0000 7f287435b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c106410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.282+0000 7f287435b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c106950 con 0x7f286c106ab0 2023-07-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.283+0000 7f28720f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c106410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.283+0000 7f28720f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c106410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35186/0 (socket says 172.21.15.49:35186) 2023-07-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.283+0000 7f28720f7700 1 -- 172.21.15.49:0/3493491916 learned_addr learned my addr 172.21.15.49:0/3493491916 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.283+0000 7f28720f7700 1 -- 172.21.15.49:0/3493491916 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868009550 con 0x7f286c106ab0 2023-07-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.283+0000 7f28720f7700 1 --2- 172.21.15.49:0/3493491916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c106410 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2868009130 tx=0x7f2868011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.283+0000 7f28637fe700 1 -- 172.21.15.49:0/3493491916 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2868003b70 con 0x7f286c106ab0 2023-07-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.283+0000 7f28637fe700 1 -- 172.21.15.49:0/3493491916 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2868003cd0 con 0x7f286c106ab0 2023-07-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.283+0000 7f287435b700 1 -- 172.21.15.49:0/3493491916 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286c104a60 con 0x7f286c106ab0 2023-07-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.283+0000 7f28637fe700 1 -- 172.21.15.49:0/3493491916 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28680197d0 con 0x7f286c106ab0 2023-07-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.283+0000 7f287435b700 1 -- 172.21.15.49:0/3493491916 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c104e80 con 0x7f286c106ab0 2023-07-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.284+0000 7f28637fe700 1 -- 172.21.15.49:0/3493491916 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 3) v1 ==== 45366+0+0 (secure 0 0 0) 0x7f286801abb0 con 0x7f286c106ab0 2023-07-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.284+0000 7f287435b700 1 -- 172.21.15.49:0/3493491916 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c10a110 con 0x7f286c106ab0 2023-07-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.284+0000 7f28637fe700 1 --2- 172.21.15.49:0/3493491916 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f28580385e0 0x7f285803aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.284+0000 7f28637fe700 1 -- 172.21.15.49:0/3493491916 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f286804c670 con 0x7f286c106ab0 2023-07-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.285+0000 7f28718f6700 1 --2- 172.21.15.49:0/3493491916 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f28580385e0 0x7f285803aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.285+0000 7f28718f6700 1 --2- 172.21.15.49:0/3493491916 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f28580385e0 0x7f285803aaa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f285c006fd0 tx=0x7f285c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.290+0000 7f28637fe700 1 -- 172.21.15.49:0/3493491916 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f286801a430 con 0x7f286c106ab0 2023-07-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.799+0000 7f287435b700 1 -- 172.21.15.49:0/3493491916 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f286c10a330 con 0x7f286c106ab0 2023-07-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.799+0000 7f28637fe700 1 -- 172.21.15.49:0/3493491916 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f2868023370 con 0x7f286c106ab0 2023-07-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.802+0000 7f287435b700 1 -- 172.21.15.49:0/3493491916 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f28580385e0 msgr2=0x7f285803aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.802+0000 7f287435b700 1 --2- 172.21.15.49:0/3493491916 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f28580385e0 0x7f285803aaa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f285c006fd0 tx=0x7f285c006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.802+0000 7f287435b700 1 -- 172.21.15.49:0/3493491916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 msgr2=0x7f286c106410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.802+0000 7f287435b700 1 --2- 172.21.15.49:0/3493491916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c106410 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2868009130 tx=0x7f2868011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.802+0000 7f287435b700 1 -- 172.21.15.49:0/3493491916 shutdown_connections 2023-07-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.802+0000 7f287435b700 1 --2- 172.21.15.49:0/3493491916 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f28580385e0 0x7f285803aaa0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.802+0000 7f287435b700 1 --2- 172.21.15.49:0/3493491916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f286c106ab0 0x7f286c106410 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.802+0000 7f287435b700 1 -- 172.21.15.49:0/3493491916 >> 172.21.15.49:0/3493491916 conn(0x7f286c100400 msgr2=0x7f286c101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.802+0000 7f287435b700 1 -- 172.21.15.49:0/3493491916 shutdown_connections 2023-07-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:14.802+0000 7f287435b700 1 -- 172.21.15.49:0/3493491916 wait complete. 2023-07-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi049.stdout:mgr is available 2023-07-20T17:42:15.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:15 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3493491916' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T17:42:16.462 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:42:16.463 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T17:42:16.463 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout fsid = 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:42:16.463 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T17:42:16.463 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T17:42:16.463 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T17:42:16.463 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T17:42:16.464 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:42:16.464 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T17:42:16.464 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T17:42:16.464 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2023-07-20T17:42:16.464 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T17:42:16.464 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T17:42:16.464 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T17:42:16.464 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T17:42:16.465 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T17:42:16.465 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T17:42:16.465 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.716+0000 7fe1cddd9700 1 Processor -- start 2023-07-20T17:42:16.465 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.716+0000 7fe1cddd9700 1 -- start start 2023-07-20T17:42:16.465 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.716+0000 7fe1cddd9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c8106890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:16.465 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.716+0000 7fe1cddd9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8106e60 con 0x7fe1c8106470 2023-07-20T17:42:16.465 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.716+0000 7fe1c77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c8106890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:16.466 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.716+0000 7fe1c77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c8106890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35190/0 (socket says 172.21.15.49:35190) 2023-07-20T17:42:16.466 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.716+0000 7fe1c77fe700 1 -- 172.21.15.49:0/2252587036 learned_addr learned my addr 172.21.15.49:0/2252587036 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:16.466 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.717+0000 7fe1c77fe700 1 -- 172.21.15.49:0/2252587036 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c8106fa0 con 0x7fe1c8106470 2023-07-20T17:42:16.466 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.717+0000 7fe1c77fe700 1 --2- 172.21.15.49:0/2252587036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c8106890 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0098a0 tx=0x7fe1bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b220a4b231db4f0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:16.466 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.717+0000 7fe1c67fc700 1 -- 172.21.15.49:0/2252587036 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc004160 con 0x7fe1c8106470 2023-07-20T17:42:16.466 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.717+0000 7fe1c67fc700 1 -- 172.21.15.49:0/2252587036 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1bc0042c0 con 0x7fe1c8106470 2023-07-20T17:42:16.466 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.717+0000 7fe1c67fc700 1 -- 172.21.15.49:0/2252587036 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc011a00 con 0x7fe1c8106470 2023-07-20T17:42:16.466 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.718+0000 7fe1cddd9700 1 -- 172.21.15.49:0/2252587036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 msgr2=0x7fe1c8106890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:16.467 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.718+0000 7fe1cddd9700 1 --2- 172.21.15.49:0/2252587036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c8106890 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0098a0 tx=0x7fe1bc009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:16.467 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.718+0000 7fe1cddd9700 1 -- 172.21.15.49:0/2252587036 shutdown_connections 2023-07-20T17:42:16.467 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.718+0000 7fe1cddd9700 1 --2- 172.21.15.49:0/2252587036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c8106890 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:16.467 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.718+0000 7fe1cddd9700 1 -- 172.21.15.49:0/2252587036 >> 172.21.15.49:0/2252587036 conn(0x7fe1c80ffbc0 msgr2=0x7fe1c8101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:16.467 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.718+0000 7fe1cddd9700 1 -- 172.21.15.49:0/2252587036 shutdown_connections 2023-07-20T17:42:16.467 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.718+0000 7fe1cddd9700 1 -- 172.21.15.49:0/2252587036 wait complete. 2023-07-20T17:42:16.467 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.718+0000 7fe1cddd9700 1 Processor -- start 2023-07-20T17:42:16.468 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1cddd9700 1 -- start start 2023-07-20T17:42:16.468 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1cddd9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c819be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:16.468 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1cddd9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c819c390 con 0x7fe1c8106470 2023-07-20T17:42:16.468 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1c77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c819be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:16.468 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1c77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c819be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35192/0 (socket says 172.21.15.49:35192) 2023-07-20T17:42:16.468 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1c77fe700 1 -- 172.21.15.49:0/1235883247 learned_addr learned my addr 172.21.15.49:0/1235883247 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:16.468 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1c77fe700 1 -- 172.21.15.49:0/1235883247 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc009550 con 0x7fe1c8106470 2023-07-20T17:42:16.469 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1c77fe700 1 --2- 172.21.15.49:0/1235883247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c819be50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009130 tx=0x7fe1bc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:16.469 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1c4ff9700 1 -- 172.21.15.49:0/1235883247 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc003ab0 con 0x7fe1c8106470 2023-07-20T17:42:16.469 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1c4ff9700 1 -- 172.21.15.49:0/1235883247 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1bc003c10 con 0x7fe1c8106470 2023-07-20T17:42:16.469 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1c4ff9700 1 -- 172.21.15.49:0/1235883247 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc0247d0 con 0x7fe1c8106470 2023-07-20T17:42:16.469 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.719+0000 7fe1cddd9700 1 -- 172.21.15.49:0/1235883247 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c819c590 con 0x7fe1c8106470 2023-07-20T17:42:16.469 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.720+0000 7fe1cddd9700 1 -- 172.21.15.49:0/1235883247 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c819c9b0 con 0x7fe1c8106470 2023-07-20T17:42:16.469 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.720+0000 7fe1c4ff9700 1 -- 172.21.15.49:0/1235883247 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fe1bc02d3f0 con 0x7fe1c8106470 2023-07-20T17:42:16.469 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.720+0000 7fe1c4ff9700 1 --2- 172.21.15.49:0/1235883247 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fe1b00386c0 0x7fe1b003ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:16.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.721+0000 7fe1c4ff9700 1 -- 172.21.15.49:0/1235883247 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe1bc04c4e0 con 0x7fe1c8106470 2023-07-20T17:42:16.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.721+0000 7fe1cddd9700 1 -- 172.21.15.49:0/1235883247 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c8195b60 con 0x7fe1c8106470 2023-07-20T17:42:16.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.721+0000 7fe1c6ffd700 1 --2- 172.21.15.49:0/1235883247 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fe1b00386c0 0x7fe1b003ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:16.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.721+0000 7fe1c6ffd700 1 --2- 172.21.15.49:0/1235883247 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fe1b00386c0 0x7fe1b003ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8006fd0 tx=0x7fe1b8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:16.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:15.725+0000 7fe1c4ff9700 1 -- 172.21.15.49:0/1235883247 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe1bc010970 con 0x7fe1c8106470 2023-07-20T17:42:16.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.080+0000 7fe1cddd9700 1 -- 172.21.15.49:0/1235883247 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe1c8061da0 con 0x7fe1c8106470 2023-07-20T17:42:16.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.080+0000 7fe1c4ff9700 1 -- 172.21.15.49:0/1235883247 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fe1bc020220 con 0x7fe1c8106470 2023-07-20T17:42:16.470 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.083+0000 7fe1cddd9700 1 -- 172.21.15.49:0/1235883247 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fe1b00386c0 msgr2=0x7fe1b003ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:16.471 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.083+0000 7fe1cddd9700 1 --2- 172.21.15.49:0/1235883247 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fe1b00386c0 0x7fe1b003ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8006fd0 tx=0x7fe1b8006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:16.471 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.083+0000 7fe1cddd9700 1 -- 172.21.15.49:0/1235883247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 msgr2=0x7fe1c819be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:16.471 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.083+0000 7fe1cddd9700 1 --2- 172.21.15.49:0/1235883247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c819be50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009130 tx=0x7fe1bc011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:16.471 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.083+0000 7fe1cddd9700 1 -- 172.21.15.49:0/1235883247 shutdown_connections 2023-07-20T17:42:16.471 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.083+0000 7fe1cddd9700 1 --2- 172.21.15.49:0/1235883247 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fe1b00386c0 0x7fe1b003ab80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:16.471 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.083+0000 7fe1cddd9700 1 --2- 172.21.15.49:0/1235883247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1c8106470 0x7fe1c819be50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:16.471 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.083+0000 7fe1cddd9700 1 -- 172.21.15.49:0/1235883247 >> 172.21.15.49:0/1235883247 conn(0x7fe1c80ffbc0 msgr2=0x7fe1c8192ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:16.471 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.083+0000 7fe1cddd9700 1 -- 172.21.15.49:0/1235883247 shutdown_connections 2023-07-20T17:42:16.472 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.083+0000 7fe1cddd9700 1 -- 172.21.15.49:0/1235883247 wait complete. 2023-07-20T17:42:16.472 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling cephadm module... 2023-07-20T17:42:16.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:16 smithi049 ceph-mon[96530]: mgrmap e4: x(active, since 3s) 2023-07-20T17:42:16.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:16 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1235883247' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T17:42:18.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:18 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1313290534' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-20T17:42:18.610 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:18 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: ignoring --setuser ceph since I am not root 2023-07-20T17:42:18.610 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:18 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: ignoring --setgroup ceph since I am not root 2023-07-20T17:42:18.610 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:18 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:18.513+0000 7f778fa60200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T17:42:18.709 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.955+0000 7f6755e28700 1 Processor -- start 2023-07-20T17:42:18.710 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.955+0000 7f6755e28700 1 -- start start 2023-07-20T17:42:18.710 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.955+0000 7f6755e28700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f6750105290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:18.710 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.955+0000 7f6755e28700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67501057d0 con 0x7f6750104e70 2023-07-20T17:42:18.710 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.955+0000 7f674f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f6750105290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:18.710 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.955+0000 7f674f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f6750105290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35196/0 (socket says 172.21.15.49:35196) 2023-07-20T17:42:18.710 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.955+0000 7f674f7fe700 1 -- 172.21.15.49:0/415164564 learned_addr learned my addr 172.21.15.49:0/415164564 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:18.710 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.956+0000 7f674f7fe700 1 -- 172.21.15.49:0/415164564 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6750105910 con 0x7f6750104e70 2023-07-20T17:42:18.711 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.956+0000 7f674f7fe700 1 --2- 172.21.15.49:0/415164564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f6750105290 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f67400098a0 tx=0x7f6740009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808a164c5155a55c server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:18.711 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.956+0000 7f674e7fc700 1 -- 172.21.15.49:0/415164564 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6740004160 con 0x7f6750104e70 2023-07-20T17:42:18.711 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.956+0000 7f674e7fc700 1 -- 172.21.15.49:0/415164564 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67400042c0 con 0x7f6750104e70 2023-07-20T17:42:18.711 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.956+0000 7f674e7fc700 1 -- 172.21.15.49:0/415164564 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6740011a00 con 0x7f6750104e70 2023-07-20T17:42:18.711 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.957+0000 7f6755e28700 1 -- 172.21.15.49:0/415164564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 msgr2=0x7f6750105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:18.711 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.957+0000 7f6755e28700 1 --2- 172.21.15.49:0/415164564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f6750105290 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f67400098a0 tx=0x7f6740009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:18.711 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.957+0000 7f6755e28700 1 -- 172.21.15.49:0/415164564 shutdown_connections 2023-07-20T17:42:18.711 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.957+0000 7f6755e28700 1 --2- 172.21.15.49:0/415164564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f6750105290 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:18.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.957+0000 7f6755e28700 1 -- 172.21.15.49:0/415164564 >> 172.21.15.49:0/415164564 conn(0x7f6750100430 msgr2=0x7f6750102850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:18.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.957+0000 7f6755e28700 1 -- 172.21.15.49:0/415164564 shutdown_connections 2023-07-20T17:42:18.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.957+0000 7f6755e28700 1 -- 172.21.15.49:0/415164564 wait complete. 2023-07-20T17:42:18.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.957+0000 7f6755e28700 1 Processor -- start 2023-07-20T17:42:18.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f6755e28700 1 -- start start 2023-07-20T17:42:18.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f6755e28700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f675018ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:18.712 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f6755e28700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67501904b0 con 0x7f6750104e70 2023-07-20T17:42:18.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f674f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f675018ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:18.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f674f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f675018ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35198/0 (socket says 172.21.15.49:35198) 2023-07-20T17:42:18.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f674f7fe700 1 -- 172.21.15.49:0/1313290534 learned_addr learned my addr 172.21.15.49:0/1313290534 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:18.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f674f7fe700 1 -- 172.21.15.49:0/1313290534 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6740009550 con 0x7f6750104e70 2023-07-20T17:42:18.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f674f7fe700 1 --2- 172.21.15.49:0/1313290534 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f675018ff70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6740009fb0 tx=0x7f6740011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:18.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f674cff9700 1 -- 172.21.15.49:0/1313290534 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6740003a40 con 0x7f6750104e70 2023-07-20T17:42:18.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f674cff9700 1 -- 172.21.15.49:0/1313290534 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6740003ba0 con 0x7f6750104e70 2023-07-20T17:42:18.713 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f674cff9700 1 -- 172.21.15.49:0/1313290534 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f674001a750 con 0x7f6750104e70 2023-07-20T17:42:18.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f6755e28700 1 -- 172.21.15.49:0/1313290534 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675018e5c0 con 0x7f6750104e70 2023-07-20T17:42:18.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.958+0000 7f6755e28700 1 -- 172.21.15.49:0/1313290534 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675018e9e0 con 0x7f6750104e70 2023-07-20T17:42:18.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.959+0000 7f674cff9700 1 -- 172.21.15.49:0/1313290534 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f6740024b60 con 0x7f6750104e70 2023-07-20T17:42:18.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.959+0000 7f674cff9700 1 --2- 172.21.15.49:0/1313290534 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f67380386c0 0x7f673803ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:18.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.960+0000 7f674cff9700 1 -- 172.21.15.49:0/1313290534 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f674004c330 con 0x7f6750104e70 2023-07-20T17:42:18.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.960+0000 7f674effd700 1 --2- 172.21.15.49:0/1313290534 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f67380386c0 0x7f673803ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:18.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.960+0000 7f6755e28700 1 -- 172.21.15.49:0/1313290534 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6750061da0 con 0x7f6750104e70 2023-07-20T17:42:18.714 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.960+0000 7f674effd700 1 --2- 172.21.15.49:0/1313290534 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f67380386c0 0x7f673803ab80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6744006fd0 tx=0x7f6744006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:18.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:16.964+0000 7f674cff9700 1 -- 172.21.15.49:0/1313290534 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f674002d3f0 con 0x7f6750104e70 2023-07-20T17:42:18.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:17.433+0000 7f6755e28700 1 -- 172.21.15.49:0/1313290534 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f6750106ca0 con 0x7f6750104e70 2023-07-20T17:42:18.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.320+0000 7f674cff9700 1 -- 172.21.15.49:0/1313290534 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f674002d770 con 0x7f6750104e70 2023-07-20T17:42:18.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.322+0000 7f674cff9700 1 -- 172.21.15.49:0/1313290534 <== mon.0 v2:172.21.15.49:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f674004c090 con 0x7f6750104e70 2023-07-20T17:42:18.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.325+0000 7f6755e28700 1 -- 172.21.15.49:0/1313290534 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f67380386c0 msgr2=0x7f673803ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:18.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.325+0000 7f6755e28700 1 --2- 172.21.15.49:0/1313290534 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f67380386c0 0x7f673803ab80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6744006fd0 tx=0x7f6744006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:18.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.325+0000 7f6755e28700 1 -- 172.21.15.49:0/1313290534 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 msgr2=0x7f675018ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:18.715 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.325+0000 7f6755e28700 1 --2- 172.21.15.49:0/1313290534 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f675018ff70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6740009fb0 tx=0x7f6740011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:18.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.325+0000 7f6755e28700 1 -- 172.21.15.49:0/1313290534 shutdown_connections 2023-07-20T17:42:18.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.325+0000 7f6755e28700 1 --2- 172.21.15.49:0/1313290534 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f67380386c0 0x7f673803ab80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:18.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.325+0000 7f6755e28700 1 --2- 172.21.15.49:0/1313290534 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6750104e70 0x7f675018ff70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:18.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.325+0000 7f6755e28700 1 -- 172.21.15.49:0/1313290534 >> 172.21.15.49:0/1313290534 conn(0x7f6750100430 msgr2=0x7f6750079960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:18.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.325+0000 7f6755e28700 1 -- 172.21.15.49:0/1313290534 shutdown_connections 2023-07-20T17:42:18.716 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:18.325+0000 7f6755e28700 1 -- 172.21.15.49:0/1313290534 wait complete. 2023-07-20T17:42:18.930 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:18 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:18.609+0000 7f778fa60200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T17:42:18.931 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:18 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:18.720+0000 7f778fa60200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T17:42:18.931 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:18 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:18.837+0000 7f778fa60200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T17:42:19.237 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:19 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:19.050+0000 7f778fa60200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T17:42:19.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:19 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1313290534' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-20T17:42:19.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:19 smithi049 ceph-mon[96530]: mgrmap e5: x(active, since 5s) 2023-07-20T17:42:19.717 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:19 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:19.457+0000 7f778fa60200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T17:42:19.717 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:19 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:19.565+0000 7f778fa60200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T17:42:19.987 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:19 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:19.718+0000 7f778fa60200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T17:42:19.988 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:19 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:19.845+0000 7f778fa60200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T17:42:20.155 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2023-07-20T17:42:20.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-20T17:42:20.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T17:42:20.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T17:42:20.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T17:42:20.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2023-07-20T17:42:20.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.245+0000 7f87163d9700 1 Processor -- start 2023-07-20T17:42:20.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.245+0000 7f87163d9700 1 -- start start 2023-07-20T17:42:20.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.245+0000 7f87163d9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f8710105280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:20.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.245+0000 7f87163d9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101057c0 con 0x7f8710104e60 2023-07-20T17:42:20.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.245+0000 7f870ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f8710105280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:20.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.245+0000 7f870ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f8710105280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35212/0 (socket says 172.21.15.49:35212) 2023-07-20T17:42:20.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.245+0000 7f870ffff700 1 -- 172.21.15.49:0/1749698323 learned_addr learned my addr 172.21.15.49:0/1749698323 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:20.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.246+0000 7f870ffff700 1 -- 172.21.15.49:0/1749698323 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8710105900 con 0x7f8710104e60 2023-07-20T17:42:20.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.246+0000 7f870ffff700 1 --2- 172.21.15.49:0/1749698323 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f8710105280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f870000bb40 tx=0x7f870000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf07713ca74c8d7 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:20.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.246+0000 7f870effd700 1 -- 172.21.15.49:0/1749698323 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f870000dbe0 con 0x7f8710104e60 2023-07-20T17:42:20.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.246+0000 7f870effd700 1 -- 172.21.15.49:0/1749698323 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87000044d0 con 0x7f8710104e60 2023-07-20T17:42:20.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.246+0000 7f870effd700 1 -- 172.21.15.49:0/1749698323 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8700010470 con 0x7f8710104e60 2023-07-20T17:42:20.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.247+0000 7f87163d9700 1 -- 172.21.15.49:0/1749698323 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 msgr2=0x7f8710105280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:20.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.247+0000 7f87163d9700 1 --2- 172.21.15.49:0/1749698323 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f8710105280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f870000bb40 tx=0x7f870000be50 comp rx=0 tx=0).stop 2023-07-20T17:42:20.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.247+0000 7f87163d9700 1 -- 172.21.15.49:0/1749698323 shutdown_connections 2023-07-20T17:42:20.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.247+0000 7f87163d9700 1 --2- 172.21.15.49:0/1749698323 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f8710105280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:20.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.247+0000 7f87163d9700 1 -- 172.21.15.49:0/1749698323 >> 172.21.15.49:0/1749698323 conn(0x7f8710100400 msgr2=0x7f8710102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:20.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.247+0000 7f87163d9700 1 -- 172.21.15.49:0/1749698323 shutdown_connections 2023-07-20T17:42:20.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.247+0000 7f87163d9700 1 -- 172.21.15.49:0/1749698323 wait complete. 2023-07-20T17:42:20.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.247+0000 7f87163d9700 1 Processor -- start 2023-07-20T17:42:20.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.247+0000 7f87163d9700 1 -- start start 2023-07-20T17:42:20.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.248+0000 7f87163d9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f871019be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:20.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.248+0000 7f87163d9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871019c340 con 0x7f8710104e60 2023-07-20T17:42:20.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.248+0000 7f870ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f871019be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:20.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.248+0000 7f870ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f871019be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35214/0 (socket says 172.21.15.49:35214) 2023-07-20T17:42:20.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.248+0000 7f870ffff700 1 -- 172.21.15.49:0/1120434500 learned_addr learned my addr 172.21.15.49:0/1120434500 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:20.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.248+0000 7f870ffff700 1 -- 172.21.15.49:0/1120434500 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870000b7f0 con 0x7f8710104e60 2023-07-20T17:42:20.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.248+0000 7f870ffff700 1 --2- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f871019be00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8700003b20 tx=0x7f8700003980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:20.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.248+0000 7f870d7fa700 1 -- 172.21.15.49:0/1120434500 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8700004340 con 0x7f8710104e60 2023-07-20T17:42:20.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.248+0000 7f870d7fa700 1 -- 172.21.15.49:0/1120434500 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8700010b80 con 0x7f8710104e60 2023-07-20T17:42:20.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.248+0000 7f87163d9700 1 -- 172.21.15.49:0/1120434500 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871019c540 con 0x7f8710104e60 2023-07-20T17:42:20.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.249+0000 7f870d7fa700 1 -- 172.21.15.49:0/1120434500 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8700021bf0 con 0x7f8710104e60 2023-07-20T17:42:20.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.249+0000 7f87163d9700 1 -- 172.21.15.49:0/1120434500 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871019c900 con 0x7f8710104e60 2023-07-20T17:42:20.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.249+0000 7f870d7fa700 1 -- 172.21.15.49:0/1120434500 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f87000193f0 con 0x7f8710104e60 2023-07-20T17:42:20.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.250+0000 7f870d7fa700 1 --2- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f86f8038760 0x7f86f803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:20.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.250+0000 7f870d7fa700 1 -- 172.21.15.49:0/1120434500 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f870004c640 con 0x7f8710104e60 2023-07-20T17:42:20.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.250+0000 7f870f7fe700 1 -- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f86f8038760 msgr2=0x7f86f803ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/3017374204 2023-07-20T17:42:20.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.250+0000 7f870f7fe700 1 --2- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f86f8038760 0x7f86f803ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T17:42:20.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.250+0000 7f87163d9700 1 -- 172.21.15.49:0/1120434500 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87101916e0 con 0x7f8710104e60 2023-07-20T17:42:20.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.255+0000 7f870d7fa700 1 -- 172.21.15.49:0/1120434500 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8700021e60 con 0x7f8710104e60 2023-07-20T17:42:20.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.451+0000 7f870f7fe700 1 -- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f86f8038760 msgr2=0x7f86f803ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/3017374204 2023-07-20T17:42:20.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.451+0000 7f870f7fe700 1 --2- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f86f8038760 0x7f86f803ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T17:42:20.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.728+0000 7f87163d9700 1 -- 172.21.15.49:0/1120434500 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f8710106d90 con 0x7f8710104e60 2023-07-20T17:42:20.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.728+0000 7f870d7fa700 1 -- 172.21.15.49:0/1120434500 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f870004a070 con 0x7f8710104e60 2023-07-20T17:42:20.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.731+0000 7f87163d9700 1 -- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f86f8038760 msgr2=0x7f86f803ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T17:42:20.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.731+0000 7f87163d9700 1 --2- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f86f8038760 0x7f86f803ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:20.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.731+0000 7f87163d9700 1 -- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 msgr2=0x7f871019be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:20.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.731+0000 7f87163d9700 1 --2- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f871019be00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8700003b20 tx=0x7f8700003980 comp rx=0 tx=0).stop 2023-07-20T17:42:20.163 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.731+0000 7f87163d9700 1 -- 172.21.15.49:0/1120434500 shutdown_connections 2023-07-20T17:42:20.163 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.731+0000 7f87163d9700 1 --2- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7f86f8038760 0x7f86f803ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:20.163 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.731+0000 7f87163d9700 1 --2- 172.21.15.49:0/1120434500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8710104e60 0x7f871019be00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:20.163 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.731+0000 7f87163d9700 1 -- 172.21.15.49:0/1120434500 >> 172.21.15.49:0/1120434500 conn(0x7f8710100400 msgr2=0x7f871018e820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:20.163 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.731+0000 7f87163d9700 1 -- 172.21.15.49:0/1120434500 shutdown_connections 2023-07-20T17:42:20.163 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:19.731+0000 7f87163d9700 1 -- 172.21.15.49:0/1120434500 wait complete. 2023-07-20T17:42:20.163 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for the mgr to restart... 2023-07-20T17:42:20.163 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr epoch 5... 2023-07-20T17:42:20.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:20 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1120434500' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T17:42:20.737 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:20 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:20.364+0000 7f778fa60200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T17:42:21.487 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:21 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:21.058+0000 7f778fa60200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T17:42:21.488 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:21 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:21.248+0000 7f778fa60200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T17:42:22.487 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:22 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:22.211+0000 7f778fa60200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T17:42:22.488 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:22 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:22.323+0000 7f778fa60200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T17:42:22.887 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:22 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:22.541+0000 7f778fa60200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T17:42:23.163 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:22 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:22.888+0000 7f778fa60200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T17:42:23.487 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:23 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:23.164+0000 7f778fa60200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T17:42:23.488 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:23 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:23.346+0000 7f778fa60200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T17:42:23.987 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:23 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:23.670+0000 7f778fa60200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T17:42:24.892 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:24 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:24.492+0000 7f778fa60200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T17:42:24.893 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:24 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:24.650+0000 7f778fa60200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T17:42:25.237 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:24 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:24.892+0000 7f778fa60200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T17:42:25.238 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:25 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:25.029+0000 7f778fa60200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T17:42:25.737 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:25 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:25.299+0000 7f778fa60200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T17:42:26.737 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:26.337+0000 7f778fa60200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T17:42:26.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: Active manager daemon x restarted 2023-07-20T17:42:26.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: Activating manager daemon x 2023-07-20T17:42:26.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: osdmap e2: 0 total, 0 up, 0 in 2023-07-20T17:42:26.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: mgrmap e6: x(active, starting, since 0.00315281s) 2023-07-20T17:42:26.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:42:26.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T17:42:26.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T17:42:26.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T17:42:26.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T17:42:26.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: Manager daemon x is now available 2023-07-20T17:42:26.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:26.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:26.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:42:26.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:26 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:42:27.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:27 smithi049 ceph-mon[96530]: Found migration_current of "None". Setting to last migration. 2023-07-20T17:42:27.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:27 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:42:27.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:27 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T17:42:27.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:27 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T17:42:27.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:27 smithi049 ceph-mon[96530]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T17:42:27.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:27 smithi049 ceph-mon[96530]: mgrmap e7: x(active, since 1.00614s) 2023-07-20T17:42:27.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:27 smithi049 ceph-mon[96530]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T17:42:27.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2023-07-20T17:42:27.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-20T17:42:27.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T17:42:27.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2023-07-20T17:42:27.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.721+0000 7fb00a422700 1 Processor -- start 2023-07-20T17:42:27.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.722+0000 7fb00a422700 1 -- start start 2023-07-20T17:42:27.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.722+0000 7fb00a422700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004108e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:27.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.722+0000 7fb00a422700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb004079e70 con 0x7fb004106a10 2023-07-20T17:42:27.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.722+0000 7fb003fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004108e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:27.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.722+0000 7fb003fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004108e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35222/0 (socket says 172.21.15.49:35222) 2023-07-20T17:42:27.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.722+0000 7fb003fff700 1 -- 172.21.15.49:0/2175379779 learned_addr learned my addr 172.21.15.49:0/2175379779 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:27.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.722+0000 7fb003fff700 1 -- 172.21.15.49:0/2175379779 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb004109340 con 0x7fb004106a10 2023-07-20T17:42:27.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.722+0000 7fb003fff700 1 --2- 172.21.15.49:0/2175379779 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004108e00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faff40098a0 tx=0x7faff4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbac3401fe940ed server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:27.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.723+0000 7fb002ffd700 1 -- 172.21.15.49:0/2175379779 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faff4004160 con 0x7fb004106a10 2023-07-20T17:42:27.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.723+0000 7fb002ffd700 1 -- 172.21.15.49:0/2175379779 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faff40042c0 con 0x7fb004106a10 2023-07-20T17:42:27.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.723+0000 7fb002ffd700 1 -- 172.21.15.49:0/2175379779 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faff4011a00 con 0x7fb004106a10 2023-07-20T17:42:27.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.724+0000 7fb00a422700 1 -- 172.21.15.49:0/2175379779 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 msgr2=0x7fb004108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:27.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.724+0000 7fb00a422700 1 --2- 172.21.15.49:0/2175379779 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004108e00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faff40098a0 tx=0x7faff4009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:27.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.724+0000 7fb00a422700 1 -- 172.21.15.49:0/2175379779 shutdown_connections 2023-07-20T17:42:27.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.724+0000 7fb00a422700 1 --2- 172.21.15.49:0/2175379779 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004108e00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:27.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.724+0000 7fb00a422700 1 -- 172.21.15.49:0/2175379779 >> 172.21.15.49:0/2175379779 conn(0x7fb004100400 msgr2=0x7fb004102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:27.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.724+0000 7fb00a422700 1 -- 172.21.15.49:0/2175379779 shutdown_connections 2023-07-20T17:42:27.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.724+0000 7fb00a422700 1 -- 172.21.15.49:0/2175379779 wait complete. 2023-07-20T17:42:27.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.725+0000 7fb00a422700 1 Processor -- start 2023-07-20T17:42:27.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.725+0000 7fb00a422700 1 -- start start 2023-07-20T17:42:27.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.725+0000 7fb00a422700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004106370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:27.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.725+0000 7fb00a422700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0041068b0 con 0x7fb004106a10 2023-07-20T17:42:27.820 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.725+0000 7fb003fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004106370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:27.820 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.725+0000 7fb003fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004106370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35224/0 (socket says 172.21.15.49:35224) 2023-07-20T17:42:27.820 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.725+0000 7fb003fff700 1 -- 172.21.15.49:0/1065574168 learned_addr learned my addr 172.21.15.49:0/1065574168 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:27.820 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.725+0000 7fb003fff700 1 -- 172.21.15.49:0/1065574168 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff4009550 con 0x7fb004106a10 2023-07-20T17:42:27.820 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.726+0000 7fb003fff700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004106370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faff4009fb0 tx=0x7faff4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:27.820 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.726+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faff4003a40 con 0x7fb004106a10 2023-07-20T17:42:27.820 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.726+0000 7fb00a422700 1 -- 172.21.15.49:0/1065574168 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0041049c0 con 0x7fb004106a10 2023-07-20T17:42:27.820 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.726+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faff4003ba0 con 0x7fb004106a10 2023-07-20T17:42:27.821 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.726+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faff401a750 con 0x7fb004106a10 2023-07-20T17:42:27.821 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.726+0000 7fb00a422700 1 -- 172.21.15.49:0/1065574168 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb004104de0 con 0x7fb004106a10 2023-07-20T17:42:27.821 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.727+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7faff401a9d0 con 0x7fb004106a10 2023-07-20T17:42:27.821 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.727+0000 7fb0017fa700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 0x7fafec03ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:27.821 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.727+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 --> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafec03b290 con 0x7fafec0386c0 2023-07-20T17:42:27.821 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.727+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7faff404c3d0 con 0x7fb004106a10 2023-07-20T17:42:27.821 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.727+0000 7fb0037fe700 1 -- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 msgr2=0x7fafec03ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/3017374204 2023-07-20T17:42:27.822 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.727+0000 7fb0037fe700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 0x7fafec03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T17:42:27.822 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.928+0000 7fb0037fe700 1 -- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 msgr2=0x7fafec03ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/3017374204 2023-07-20T17:42:27.822 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:20.928+0000 7fb0037fe700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 0x7fafec03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T17:42:27.822 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:21.329+0000 7fb0037fe700 1 -- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 msgr2=0x7fafec03ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/3017374204 2023-07-20T17:42:27.822 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:21.329+0000 7fb0037fe700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 0x7fafec03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T17:42:27.822 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:22.130+0000 7fb0037fe700 1 -- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 msgr2=0x7fafec03ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/3017374204 2023-07-20T17:42:27.822 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:22.130+0000 7fb0037fe700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 0x7fafec03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T17:42:27.823 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:23.730+0000 7fb0037fe700 1 -- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 msgr2=0x7fafec03ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/3017374204 2023-07-20T17:42:27.823 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:23.730+0000 7fb0037fe700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 0x7fafec03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T17:42:27.823 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:26.352+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7faff402d9b0 con 0x7fb004106a10 2023-07-20T17:42:27.823 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:26.352+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 msgr2=0x7fafec03ab80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T17:42:27.823 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:26.352+0000 7fb0017fa700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 0x7fafec03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:27.823 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.353+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7faff404d9e0 con 0x7fb004106a10 2023-07-20T17:42:27.823 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.354+0000 7fb0017fa700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fafec03bff0 0x7fafec03e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:27.823 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.354+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 --> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafec03b290 con 0x7fafec03bff0 2023-07-20T17:42:27.824 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.354+0000 7fb0037fe700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fafec03bff0 0x7fafec03e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:27.824 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.355+0000 7fb0037fe700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fafec03bff0 0x7fafec03e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7faff8003a10 tx=0x7faff8009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:27.824 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.355+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 <== mgr.14120 v2:172.21.15.49:6800/752420140 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fafec03b290 con 0x7fafec03bff0 2023-07-20T17:42:27.824 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.366+0000 7fb00a422700 1 -- 172.21.15.49:0/1065574168 --> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb004105f10 con 0x7fafec03bff0 2023-07-20T17:42:27.824 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.366+0000 7fb0017fa700 1 -- 172.21.15.49:0/1065574168 <== mgr.14120 v2:172.21.15.49:6800/752420140 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb004105f10 con 0x7fafec03bff0 2023-07-20T17:42:27.824 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.367+0000 7fb00a422700 1 -- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fafec03bff0 msgr2=0x7fafec03e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:27.824 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.367+0000 7fb00a422700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fafec03bff0 0x7fafec03e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7faff8003a10 tx=0x7faff8009290 comp rx=0 tx=0).stop 2023-07-20T17:42:27.824 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.367+0000 7fb00a422700 1 -- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 msgr2=0x7fb004106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:27.825 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.367+0000 7fb00a422700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004106370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faff4009fb0 tx=0x7faff4011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:27.825 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.367+0000 7fb00a422700 1 -- 172.21.15.49:0/1065574168 shutdown_connections 2023-07-20T17:42:27.825 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.367+0000 7fb00a422700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fafec03bff0 0x7fafec03e3e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:27.825 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.367+0000 7fb00a422700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb004106a10 0x7fb004106370 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:27.825 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.367+0000 7fb00a422700 1 --2- 172.21.15.49:0/1065574168 >> [v2:172.21.15.49:6800/3017374204,v1:172.21.15.49:6801/3017374204] conn(0x7fafec0386c0 0x7fafec03ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:27.825 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.367+0000 7fb00a422700 1 -- 172.21.15.49:0/1065574168 >> 172.21.15.49:0/1065574168 conn(0x7fb004100400 msgr2=0x7fb004101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:27.825 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.367+0000 7fb00a422700 1 -- 172.21.15.49:0/1065574168 shutdown_connections 2023-07-20T17:42:27.826 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:27.367+0000 7fb00a422700 1 -- 172.21.15.49:0/1065574168 wait complete. 2023-07-20T17:42:27.826 INFO:teuthology.orchestra.run.smithi049.stdout:mgr epoch 5 is available 2023-07-20T17:42:27.826 INFO:teuthology.orchestra.run.smithi049.stdout:Setting orchestrator backend to cephadm... 2023-07-20T17:42:28.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:28 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:28.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:28 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:28.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:28 smithi049 ceph-mon[96530]: [20/Jul/2023:17:42:28] ENGINE Bus STARTING 2023-07-20T17:42:28.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:28 smithi049 ceph-mon[96530]: [20/Jul/2023:17:42:28] ENGINE Serving on http://172.21.15.49:8765 2023-07-20T17:42:28.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:28 smithi049 ceph-mon[96530]: [20/Jul/2023:17:42:28] ENGINE Serving on https://172.21.15.49:7150 2023-07-20T17:42:28.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:28 smithi049 ceph-mon[96530]: [20/Jul/2023:17:42:28] ENGINE Bus STARTED 2023-07-20T17:42:28.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:28 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:42:28.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:28 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:29.066 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.287+0000 7f833b1a3700 1 Processor -- start 2023-07-20T17:42:29.067 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.287+0000 7f833b1a3700 1 -- start start 2023-07-20T17:42:29.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.288+0000 7f833b1a3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334108dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:29.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.288+0000 7f833b1a3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8334079eb0 con 0x7f83341069d0 2023-07-20T17:42:29.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.288+0000 7f8338f3f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334108dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:29.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.288+0000 7f8338f3f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334108dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35276/0 (socket says 172.21.15.49:35276) 2023-07-20T17:42:29.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.288+0000 7f8338f3f700 1 -- 172.21.15.49:0/2702477993 learned_addr learned my addr 172.21.15.49:0/2702477993 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:29.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.288+0000 7f8338f3f700 1 -- 172.21.15.49:0/2702477993 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8334109300 con 0x7f83341069d0 2023-07-20T17:42:29.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.288+0000 7f8338f3f700 1 --2- 172.21.15.49:0/2702477993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334108dc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f83280098a0 tx=0x7f8328009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c27b3bf2c74fc393 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:29.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.288+0000 7f83337fe700 1 -- 172.21.15.49:0/2702477993 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8328004160 con 0x7f83341069d0 2023-07-20T17:42:29.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.288+0000 7f83337fe700 1 -- 172.21.15.49:0/2702477993 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83280042c0 con 0x7f83341069d0 2023-07-20T17:42:29.072 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.289+0000 7f83337fe700 1 -- 172.21.15.49:0/2702477993 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8328011a00 con 0x7f83341069d0 2023-07-20T17:42:29.072 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.289+0000 7f833b1a3700 1 -- 172.21.15.49:0/2702477993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 msgr2=0x7f8334108dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:29.072 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.289+0000 7f833b1a3700 1 --2- 172.21.15.49:0/2702477993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334108dc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f83280098a0 tx=0x7f8328009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:29.072 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.289+0000 7f833b1a3700 1 -- 172.21.15.49:0/2702477993 shutdown_connections 2023-07-20T17:42:29.072 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.289+0000 7f833b1a3700 1 --2- 172.21.15.49:0/2702477993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334108dc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:29.072 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.289+0000 7f833b1a3700 1 -- 172.21.15.49:0/2702477993 >> 172.21.15.49:0/2702477993 conn(0x7f8334100400 msgr2=0x7f8334102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:29.073 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.289+0000 7f833b1a3700 1 -- 172.21.15.49:0/2702477993 shutdown_connections 2023-07-20T17:42:29.073 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.289+0000 7f833b1a3700 1 -- 172.21.15.49:0/2702477993 wait complete. 2023-07-20T17:42:29.073 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.290+0000 7f833b1a3700 1 Processor -- start 2023-07-20T17:42:29.073 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.290+0000 7f833b1a3700 1 -- start start 2023-07-20T17:42:29.073 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.290+0000 7f833b1a3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334106330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:29.073 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.290+0000 7f833b1a3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8334106870 con 0x7f83341069d0 2023-07-20T17:42:29.073 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.290+0000 7f8338f3f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334106330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:29.073 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.290+0000 7f8338f3f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334106330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35278/0 (socket says 172.21.15.49:35278) 2023-07-20T17:42:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.290+0000 7f8338f3f700 1 -- 172.21.15.49:0/2345309454 learned_addr learned my addr 172.21.15.49:0/2345309454 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.290+0000 7f8338f3f700 1 -- 172.21.15.49:0/2345309454 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8328009550 con 0x7f83341069d0 2023-07-20T17:42:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.290+0000 7f8338f3f700 1 --2- 172.21.15.49:0/2345309454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334106330 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8328009fb0 tx=0x7f8328011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.291+0000 7f8331ffb700 1 -- 172.21.15.49:0/2345309454 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8328003a40 con 0x7f83341069d0 2023-07-20T17:42:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.291+0000 7f8331ffb700 1 -- 172.21.15.49:0/2345309454 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8328003ba0 con 0x7f83341069d0 2023-07-20T17:42:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.291+0000 7f833b1a3700 1 -- 172.21.15.49:0/2345309454 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8334104980 con 0x7f83341069d0 2023-07-20T17:42:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.291+0000 7f8331ffb700 1 -- 172.21.15.49:0/2345309454 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f832801a750 con 0x7f83341069d0 2023-07-20T17:42:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.291+0000 7f833b1a3700 1 -- 172.21.15.49:0/2345309454 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8334104da0 con 0x7f83341069d0 2023-07-20T17:42:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.292+0000 7f8331ffb700 1 -- 172.21.15.49:0/2345309454 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f832801a980 con 0x7f83341069d0 2023-07-20T17:42:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.292+0000 7f8331ffb700 1 --2- 172.21.15.49:0/2345309454 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f83240385f0 0x7f832403aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.292+0000 7f8331ffb700 1 -- 172.21.15.49:0/2345309454 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f832804c7a0 con 0x7f83341069d0 2023-07-20T17:42:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.292+0000 7f833b1a3700 1 -- 172.21.15.49:0/2345309454 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8334061da0 con 0x7f83341069d0 2023-07-20T17:42:29.076 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.292+0000 7f8333fff700 1 --2- 172.21.15.49:0/2345309454 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f83240385f0 0x7f832403aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:29.076 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.292+0000 7f8333fff700 1 --2- 172.21.15.49:0/2345309454 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f83240385f0 0x7f832403aab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8320006fd0 tx=0x7f8320006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:29.076 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.297+0000 7f8331ffb700 1 -- 172.21.15.49:0/2345309454 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f832801ac30 con 0x7f83341069d0 2023-07-20T17:42:29.076 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.681+0000 7f833b1a3700 1 -- 172.21.15.49:0/2345309454 --> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f83341057a0 con 0x7f83240385f0 2023-07-20T17:42:29.077 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.686+0000 7f8331ffb700 1 -- 172.21.15.49:0/2345309454 <== mgr.14120 v2:172.21.15.49:6800/752420140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f83341057a0 con 0x7f83240385f0 2023-07-20T17:42:29.077 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.689+0000 7f833b1a3700 1 -- 172.21.15.49:0/2345309454 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f83240385f0 msgr2=0x7f832403aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:29.077 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.689+0000 7f833b1a3700 1 --2- 172.21.15.49:0/2345309454 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f83240385f0 0x7f832403aab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8320006fd0 tx=0x7f8320006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:29.077 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.689+0000 7f833b1a3700 1 -- 172.21.15.49:0/2345309454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 msgr2=0x7f8334106330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:29.077 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.689+0000 7f833b1a3700 1 --2- 172.21.15.49:0/2345309454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334106330 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8328009fb0 tx=0x7f8328011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:29.080 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.689+0000 7f833b1a3700 1 -- 172.21.15.49:0/2345309454 shutdown_connections 2023-07-20T17:42:29.080 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.689+0000 7f833b1a3700 1 --2- 172.21.15.49:0/2345309454 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f83240385f0 0x7f832403aab0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:29.080 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.689+0000 7f833b1a3700 1 --2- 172.21.15.49:0/2345309454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83341069d0 0x7f8334106330 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:29.081 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.689+0000 7f833b1a3700 1 -- 172.21.15.49:0/2345309454 >> 172.21.15.49:0/2345309454 conn(0x7f8334100400 msgr2=0x7f8334101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:29.081 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.689+0000 7f833b1a3700 1 -- 172.21.15.49:0/2345309454 shutdown_connections 2023-07-20T17:42:29.081 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:28.689+0000 7f833b1a3700 1 -- 172.21.15.49:0/2345309454 wait complete. 2023-07-20T17:42:30.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:29 smithi049 ceph-mon[96530]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:42:30.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:29 smithi049 ceph-mon[96530]: mgrmap e8: x(active, since 2s) 2023-07-20T17:42:30.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:29 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:42:30.315 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-20T17:42:30.315 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.548+0000 7f6397324700 1 Processor -- start 2023-07-20T17:42:30.315 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.549+0000 7f6397324700 1 -- start start 2023-07-20T17:42:30.316 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.549+0000 7f6397324700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390105280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:30.316 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.549+0000 7f6397324700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63901057c0 con 0x7f6390104e60 2023-07-20T17:42:30.316 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.549+0000 7f63950c0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390105280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:30.316 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.549+0000 7f63950c0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390105280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35282/0 (socket says 172.21.15.49:35282) 2023-07-20T17:42:30.316 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.549+0000 7f63950c0700 1 -- 172.21.15.49:0/3631513865 learned_addr learned my addr 172.21.15.49:0/3631513865 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:30.316 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.549+0000 7f63950c0700 1 -- 172.21.15.49:0/3631513865 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6390105900 con 0x7f6390104e60 2023-07-20T17:42:30.316 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.549+0000 7f63950c0700 1 --2- 172.21.15.49:0/3631513865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390105280 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f638c0098a0 tx=0x7f638c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509704cb931cde96 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:30.317 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.550+0000 7f6383fff700 1 -- 172.21.15.49:0/3631513865 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f638c004160 con 0x7f6390104e60 2023-07-20T17:42:30.317 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.550+0000 7f6383fff700 1 -- 172.21.15.49:0/3631513865 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f638c0042c0 con 0x7f6390104e60 2023-07-20T17:42:30.317 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.550+0000 7f6383fff700 1 -- 172.21.15.49:0/3631513865 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f638c011a00 con 0x7f6390104e60 2023-07-20T17:42:30.317 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.550+0000 7f6397324700 1 -- 172.21.15.49:0/3631513865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 msgr2=0x7f6390105280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:30.317 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.550+0000 7f6397324700 1 --2- 172.21.15.49:0/3631513865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390105280 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f638c0098a0 tx=0x7f638c009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:30.317 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.550+0000 7f6397324700 1 -- 172.21.15.49:0/3631513865 shutdown_connections 2023-07-20T17:42:30.317 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.550+0000 7f6397324700 1 --2- 172.21.15.49:0/3631513865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390105280 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:30.317 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.550+0000 7f6397324700 1 -- 172.21.15.49:0/3631513865 >> 172.21.15.49:0/3631513865 conn(0x7f6390100400 msgr2=0x7f6390102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:30.318 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.550+0000 7f6397324700 1 -- 172.21.15.49:0/3631513865 shutdown_connections 2023-07-20T17:42:30.318 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.550+0000 7f6397324700 1 -- 172.21.15.49:0/3631513865 wait complete. 2023-07-20T17:42:30.318 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.551+0000 7f6397324700 1 Processor -- start 2023-07-20T17:42:30.318 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.551+0000 7f6397324700 1 -- start start 2023-07-20T17:42:30.318 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.551+0000 7f6397324700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:30.318 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.551+0000 7f6397324700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6390199040 con 0x7f6390104e60 2023-07-20T17:42:30.318 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.552+0000 7f63950c0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:30.319 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.552+0000 7f63950c0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35284/0 (socket says 172.21.15.49:35284) 2023-07-20T17:42:30.319 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.552+0000 7f63950c0700 1 -- 172.21.15.49:0/3666606826 learned_addr learned my addr 172.21.15.49:0/3666606826 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:30.319 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.552+0000 7f63950c0700 1 -- 172.21.15.49:0/3666606826 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c009550 con 0x7f6390104e60 2023-07-20T17:42:30.319 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.552+0000 7f63950c0700 1 --2- 172.21.15.49:0/3666606826 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390198b00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f638c009fb0 tx=0x7f638c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:30.319 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.552+0000 7f63827fc700 1 -- 172.21.15.49:0/3666606826 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f638c003a40 con 0x7f6390104e60 2023-07-20T17:42:30.319 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.552+0000 7f63827fc700 1 -- 172.21.15.49:0/3666606826 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f638c003ba0 con 0x7f6390104e60 2023-07-20T17:42:30.319 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.552+0000 7f6397324700 1 -- 172.21.15.49:0/3666606826 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6390199240 con 0x7f6390104e60 2023-07-20T17:42:30.319 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.552+0000 7f63827fc700 1 -- 172.21.15.49:0/3666606826 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f638c01a750 con 0x7f6390104e60 2023-07-20T17:42:30.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.552+0000 7f6397324700 1 -- 172.21.15.49:0/3666606826 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6390199660 con 0x7f6390104e60 2023-07-20T17:42:30.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.553+0000 7f63827fc700 1 -- 172.21.15.49:0/3666606826 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f638c01a980 con 0x7f6390104e60 2023-07-20T17:42:30.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.553+0000 7f63827fc700 1 --2- 172.21.15.49:0/3666606826 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f637c038710 0x7f637c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:30.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.553+0000 7f63827fc700 1 -- 172.21.15.49:0/3666606826 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f638c04c950 con 0x7f6390104e60 2023-07-20T17:42:30.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.553+0000 7f6397324700 1 -- 172.21.15.49:0/3666606826 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6390061da0 con 0x7f6390104e60 2023-07-20T17:42:30.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.553+0000 7f63948bf700 1 --2- 172.21.15.49:0/3666606826 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f637c038710 0x7f637c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:30.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.554+0000 7f63948bf700 1 --2- 172.21.15.49:0/3666606826 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f637c038710 0x7f637c03abd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6384006fd0 tx=0x7f6384006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:30.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.558+0000 7f63827fc700 1 -- 172.21.15.49:0/3666606826 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f638c01ac30 con 0x7f6390104e60 2023-07-20T17:42:30.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.915+0000 7f6397324700 1 -- 172.21.15.49:0/3666606826 --> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f6390101920 con 0x7f637c038710 2023-07-20T17:42:30.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.916+0000 7f63827fc700 1 -- 172.21.15.49:0/3666606826 <== mgr.14120 v2:172.21.15.49:6800/752420140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f6390101920 con 0x7f637c038710 2023-07-20T17:42:30.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.918+0000 7f6397324700 1 -- 172.21.15.49:0/3666606826 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f637c038710 msgr2=0x7f637c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:30.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.918+0000 7f6397324700 1 --2- 172.21.15.49:0/3666606826 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f637c038710 0x7f637c03abd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6384006fd0 tx=0x7f6384006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:30.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.918+0000 7f6397324700 1 -- 172.21.15.49:0/3666606826 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 msgr2=0x7f6390198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:30.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.918+0000 7f6397324700 1 --2- 172.21.15.49:0/3666606826 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390198b00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f638c009fb0 tx=0x7f638c011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:30.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.918+0000 7f6397324700 1 -- 172.21.15.49:0/3666606826 shutdown_connections 2023-07-20T17:42:30.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.918+0000 7f6397324700 1 --2- 172.21.15.49:0/3666606826 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f637c038710 0x7f637c03abd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:30.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.918+0000 7f6397324700 1 --2- 172.21.15.49:0/3666606826 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6390104e60 0x7f6390198b00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:30.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.918+0000 7f6397324700 1 -- 172.21.15.49:0/3666606826 >> 172.21.15.49:0/3666606826 conn(0x7f6390100400 msgr2=0x7f6390101210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:30.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.918+0000 7f6397324700 1 -- 172.21.15.49:0/3666606826 shutdown_connections 2023-07-20T17:42:30.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:29.918+0000 7f6397324700 1 -- 172.21.15.49:0/3666606826 wait complete. 2023-07-20T17:42:30.322 INFO:teuthology.orchestra.run.smithi049.stdout:Generating ssh key... 2023-07-20T17:42:30.987 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:30 smithi049 ceph-mon[96530]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:42:31.860 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.801+0000 7f55f6f5e700 1 Processor -- start 2023-07-20T17:42:31.860 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.801+0000 7f55f6f5e700 1 -- start start 2023-07-20T17:42:31.860 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.801+0000 7f55f6f5e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0104680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:31.861 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.801+0000 7f55f6f5e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0104c50 con 0x7f55f0104260 2023-07-20T17:42:31.861 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.802+0000 7f55f4cfa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:31.861 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.802+0000 7f55f4cfa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0104680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35288/0 (socket says 172.21.15.49:35288) 2023-07-20T17:42:31.861 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.802+0000 7f55f4cfa700 1 -- 172.21.15.49:0/1230634197 learned_addr learned my addr 172.21.15.49:0/1230634197 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:31.861 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.802+0000 7f55f4cfa700 1 -- 172.21.15.49:0/1230634197 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f01071f0 con 0x7f55f0104260 2023-07-20T17:42:31.861 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.802+0000 7f55f4cfa700 1 --2- 172.21.15.49:0/1230634197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0104680 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f55e40098a0 tx=0x7f55e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc334fe128f0315 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:31.861 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.802+0000 7f55ef7fe700 1 -- 172.21.15.49:0/1230634197 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55e4004160 con 0x7f55f0104260 2023-07-20T17:42:31.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.802+0000 7f55ef7fe700 1 -- 172.21.15.49:0/1230634197 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55e40042c0 con 0x7f55f0104260 2023-07-20T17:42:31.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.802+0000 7f55ef7fe700 1 -- 172.21.15.49:0/1230634197 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55e4011a00 con 0x7f55f0104260 2023-07-20T17:42:31.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.803+0000 7f55f6f5e700 1 -- 172.21.15.49:0/1230634197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 msgr2=0x7f55f0104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:31.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.803+0000 7f55f6f5e700 1 --2- 172.21.15.49:0/1230634197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0104680 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f55e40098a0 tx=0x7f55e4009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:31.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.803+0000 7f55f6f5e700 1 -- 172.21.15.49:0/1230634197 shutdown_connections 2023-07-20T17:42:31.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.803+0000 7f55f6f5e700 1 --2- 172.21.15.49:0/1230634197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0104680 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:31.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.803+0000 7f55f6f5e700 1 -- 172.21.15.49:0/1230634197 >> 172.21.15.49:0/1230634197 conn(0x7f55f00ffb90 msgr2=0x7f55f0101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:31.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.803+0000 7f55f6f5e700 1 -- 172.21.15.49:0/1230634197 shutdown_connections 2023-07-20T17:42:31.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.803+0000 7f55f6f5e700 1 -- 172.21.15.49:0/1230634197 wait complete. 2023-07-20T17:42:31.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.803+0000 7f55f6f5e700 1 Processor -- start 2023-07-20T17:42:31.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.804+0000 7f55f6f5e700 1 -- start start 2023-07-20T17:42:31.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.804+0000 7f55f6f5e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0106b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:31.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.804+0000 7f55f6f5e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0107090 con 0x7f55f0104260 2023-07-20T17:42:31.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.804+0000 7f55f4cfa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0106b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:31.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.804+0000 7f55f4cfa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0106b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35290/0 (socket says 172.21.15.49:35290) 2023-07-20T17:42:31.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.804+0000 7f55f4cfa700 1 -- 172.21.15.49:0/156682244 learned_addr learned my addr 172.21.15.49:0/156682244 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:31.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.804+0000 7f55f4cfa700 1 -- 172.21.15.49:0/156682244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e4009550 con 0x7f55f0104260 2023-07-20T17:42:31.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.804+0000 7f55f4cfa700 1 --2- 172.21.15.49:0/156682244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0106b50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f55e4009130 tx=0x7f55e4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:31.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.804+0000 7f55edffb700 1 -- 172.21.15.49:0/156682244 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55e4003ab0 con 0x7f55f0104260 2023-07-20T17:42:31.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.804+0000 7f55edffb700 1 -- 172.21.15.49:0/156682244 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55e4003c10 con 0x7f55f0104260 2023-07-20T17:42:31.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.804+0000 7f55f6f5e700 1 -- 172.21.15.49:0/156682244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f01051a0 con 0x7f55f0104260 2023-07-20T17:42:31.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.805+0000 7f55edffb700 1 -- 172.21.15.49:0/156682244 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55e40247d0 con 0x7f55f0104260 2023-07-20T17:42:31.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.805+0000 7f55f6f5e700 1 -- 172.21.15.49:0/156682244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f01055c0 con 0x7f55f0104260 2023-07-20T17:42:31.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.805+0000 7f55edffb700 1 -- 172.21.15.49:0/156682244 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f55e402d3f0 con 0x7f55f0104260 2023-07-20T17:42:31.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.806+0000 7f55edffb700 1 --2- 172.21.15.49:0/156682244 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f55e0038710 0x7f55e003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:31.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.806+0000 7f55edffb700 1 -- 172.21.15.49:0/156682244 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f55e404c8b0 con 0x7f55f0104260 2023-07-20T17:42:31.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.806+0000 7f55effff700 1 --2- 172.21.15.49:0/156682244 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f55e0038710 0x7f55e003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:31.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.806+0000 7f55f6f5e700 1 -- 172.21.15.49:0/156682244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f0061da0 con 0x7f55f0104260 2023-07-20T17:42:31.866 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.806+0000 7f55effff700 1 --2- 172.21.15.49:0/156682244 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f55e0038710 0x7f55e003abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f55dc006fd0 tx=0x7f55dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:31.866 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:30.811+0000 7f55edffb700 1 -- 172.21.15.49:0/156682244 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f55e4010740 con 0x7f55f0104260 2023-07-20T17:42:31.866 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.168+0000 7f55f6f5e700 1 -- 172.21.15.49:0/156682244 --> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f55f0105fe0 con 0x7f55e0038710 2023-07-20T17:42:31.866 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.522+0000 7f55edffb700 1 -- 172.21.15.49:0/156682244 <== mgr.14120 v2:172.21.15.49:6800/752420140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f55f0105fe0 con 0x7f55e0038710 2023-07-20T17:42:31.866 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.525+0000 7f55f6f5e700 1 -- 172.21.15.49:0/156682244 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f55e0038710 msgr2=0x7f55e003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:31.866 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.525+0000 7f55f6f5e700 1 --2- 172.21.15.49:0/156682244 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f55e0038710 0x7f55e003abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f55dc006fd0 tx=0x7f55dc006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:31.866 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.525+0000 7f55f6f5e700 1 -- 172.21.15.49:0/156682244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 msgr2=0x7f55f0106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:31.866 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.525+0000 7f55f6f5e700 1 --2- 172.21.15.49:0/156682244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0106b50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f55e4009130 tx=0x7f55e4011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:31.867 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.525+0000 7f55f6f5e700 1 -- 172.21.15.49:0/156682244 shutdown_connections 2023-07-20T17:42:31.867 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.525+0000 7f55f6f5e700 1 --2- 172.21.15.49:0/156682244 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f55e0038710 0x7f55e003abd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:31.867 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.525+0000 7f55f6f5e700 1 --2- 172.21.15.49:0/156682244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0104260 0x7f55f0106b50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:31.867 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.525+0000 7f55f6f5e700 1 -- 172.21.15.49:0/156682244 >> 172.21.15.49:0/156682244 conn(0x7f55f00ffb90 msgr2=0x7f55f0101260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:31.867 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.525+0000 7f55f6f5e700 1 -- 172.21.15.49:0/156682244 shutdown_connections 2023-07-20T17:42:31.867 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:31.525+0000 7f55f6f5e700 1 -- 172.21.15.49:0/156682244 wait complete. 2023-07-20T17:42:31.885 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-mon[96530]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:42:31.885 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-mon[96530]: Generating ssh key... 2023-07-20T17:42:31.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:31.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:31.886 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: Generating public/private rsa key pair. 2023-07-20T17:42:31.886 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: Your identification has been saved in /tmp/tmp0jzk9xi8/key. 2023-07-20T17:42:31.886 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: Your public key has been saved in /tmp/tmp0jzk9xi8/key.pub. 2023-07-20T17:42:31.886 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: The key fingerprint is: 2023-07-20T17:42:31.887 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: SHA256:Cwg5iZHA3Cpbnfre7Flh6spUfctw9Wm4YFW7jsUeLgM ceph-8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:42:31.887 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: The key's randomart image is: 2023-07-20T17:42:31.887 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: +---[RSA 3072]----+ 2023-07-20T17:42:31.887 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: |*.. . | 2023-07-20T17:42:31.887 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: |.= + . . | 2023-07-20T17:42:31.887 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: |. *. . o . | 2023-07-20T17:42:31.887 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: |...oo. . o + o | 2023-07-20T17:42:31.887 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: |.o .. o S E . O | 2023-07-20T17:42:31.888 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: |. . . + O + O . | 2023-07-20T17:42:31.888 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: | .. . o o = + | 2023-07-20T17:42:31.888 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: | o.+ o o | 2023-07-20T17:42:31.888 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: | .+o* | 2023-07-20T17:42:31.888 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: +----[SHA256]-----+ 2023-07-20T17:42:33.207 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwy9yG54fHOppHe4rzNcXl5vyFil+sl+m/Cp2xfHbxUuy6+fD74k5b2iphUpGb8GoGbo9hgZRqfjNX2d+PNeDr089Cu8QaPB7oglDss1RUGlWnf0EXTLU0zuT0DwlZd24j6sDgy1ji1aIKt7vwwEXT+/zEo45GKAPoByoEHUO3f8CNg4XXgRjNaEestLxeVSlVyvDoruC45na60uKVBctm+L+b52AXMKMqivYPgbr9WRUlFQ0y3pB2rnGNN4qlUoS8Eyk9Y9gtBQoS8uE7sGtN3Safuj9M5hosHoDtnKHOoY4lq+EcbLhcR0sf8TrgSJ3zyX2m0KzOHrvTNCajBWi9a1K+RMX+C4shuHr7BJ9l7Z/smBN14wovhli+bFIQcPasEk2LGaPr3gigKSDfxEDSPSUnUVilZwRtgqtA94VwTlpE4WYX2BaR8jTIP9hbYYH0tiWLbiM7f9datpUzzjqm79eESwochrdmWf+3iasZbWEqKA6AKCOR4pCy7jb+jV0= ceph-8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:42:33.208 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.426+0000 7fd4b5e43700 1 Processor -- start 2023-07-20T17:42:33.208 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.426+0000 7fd4b5e43700 1 -- start start 2023-07-20T17:42:33.208 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.427+0000 7fd4b5e43700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b0105280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:33.208 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.427+0000 7fd4b5e43700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b01057c0 con 0x7fd4b0104e60 2023-07-20T17:42:33.208 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.427+0000 7fd4af7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b0105280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:33.208 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.427+0000 7fd4af7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b0105280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35294/0 (socket says 172.21.15.49:35294) 2023-07-20T17:42:33.209 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.427+0000 7fd4af7fe700 1 -- 172.21.15.49:0/3633508268 learned_addr learned my addr 172.21.15.49:0/3633508268 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:33.209 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.427+0000 7fd4af7fe700 1 -- 172.21.15.49:0/3633508268 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b0105900 con 0x7fd4b0104e60 2023-07-20T17:42:33.210 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.427+0000 7fd4af7fe700 1 --2- 172.21.15.49:0/3633508268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b0105280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000bb40 tx=0x7fd4a000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4450a7f10544bbe server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:33.210 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.427+0000 7fd4ae7fc700 1 -- 172.21.15.49:0/3633508268 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4a000dbe0 con 0x7fd4b0104e60 2023-07-20T17:42:33.210 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.427+0000 7fd4ae7fc700 1 -- 172.21.15.49:0/3633508268 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4a00044d0 con 0x7fd4b0104e60 2023-07-20T17:42:33.210 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.427+0000 7fd4ae7fc700 1 -- 172.21.15.49:0/3633508268 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4a0010470 con 0x7fd4b0104e60 2023-07-20T17:42:33.210 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.428+0000 7fd4b5e43700 1 -- 172.21.15.49:0/3633508268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 msgr2=0x7fd4b0105280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:33.211 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.428+0000 7fd4b5e43700 1 --2- 172.21.15.49:0/3633508268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b0105280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000bb40 tx=0x7fd4a000be50 comp rx=0 tx=0).stop 2023-07-20T17:42:33.211 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.428+0000 7fd4b5e43700 1 -- 172.21.15.49:0/3633508268 shutdown_connections 2023-07-20T17:42:33.211 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.428+0000 7fd4b5e43700 1 --2- 172.21.15.49:0/3633508268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b0105280 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:33.211 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.428+0000 7fd4b5e43700 1 -- 172.21.15.49:0/3633508268 >> 172.21.15.49:0/3633508268 conn(0x7fd4b0100400 msgr2=0x7fd4b0102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:33.211 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.428+0000 7fd4b5e43700 1 -- 172.21.15.49:0/3633508268 shutdown_connections 2023-07-20T17:42:33.211 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.428+0000 7fd4b5e43700 1 -- 172.21.15.49:0/3633508268 wait complete. 2023-07-20T17:42:33.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.429+0000 7fd4b5e43700 1 Processor -- start 2023-07-20T17:42:33.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.429+0000 7fd4b5e43700 1 -- start start 2023-07-20T17:42:33.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.429+0000 7fd4b5e43700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b01979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:33.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.429+0000 7fd4b5e43700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0197f30 con 0x7fd4b0104e60 2023-07-20T17:42:33.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.429+0000 7fd4af7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b01979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:33.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.429+0000 7fd4af7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b01979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35296/0 (socket says 172.21.15.49:35296) 2023-07-20T17:42:33.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.429+0000 7fd4af7fe700 1 -- 172.21.15.49:0/2287343302 learned_addr learned my addr 172.21.15.49:0/2287343302 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:33.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.429+0000 7fd4af7fe700 1 -- 172.21.15.49:0/2287343302 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a000b7f0 con 0x7fd4b0104e60 2023-07-20T17:42:33.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.430+0000 7fd4af7fe700 1 --2- 172.21.15.49:0/2287343302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b01979f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000d650 tx=0x7fd4a0003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:33.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.430+0000 7fd4acff9700 1 -- 172.21.15.49:0/2287343302 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4a001e070 con 0x7fd4b0104e60 2023-07-20T17:42:33.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.430+0000 7fd4b5e43700 1 -- 172.21.15.49:0/2287343302 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b0198130 con 0x7fd4b0104e60 2023-07-20T17:42:33.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.430+0000 7fd4acff9700 1 -- 172.21.15.49:0/2287343302 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4a00042a0 con 0x7fd4b0104e60 2023-07-20T17:42:33.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.430+0000 7fd4acff9700 1 -- 172.21.15.49:0/2287343302 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4a0022af0 con 0x7fd4b0104e60 2023-07-20T17:42:33.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.430+0000 7fd4b5e43700 1 -- 172.21.15.49:0/2287343302 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b0198550 con 0x7fd4b0104e60 2023-07-20T17:42:33.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.431+0000 7fd4acff9700 1 -- 172.21.15.49:0/2287343302 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fd4a002c440 con 0x7fd4b0104e60 2023-07-20T17:42:33.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.431+0000 7fd4acff9700 1 --2- 172.21.15.49:0/2287343302 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd4980386c0 0x7fd49803ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:33.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.431+0000 7fd4b5e43700 1 -- 172.21.15.49:0/2287343302 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b01916e0 con 0x7fd4b0104e60 2023-07-20T17:42:33.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.431+0000 7fd4acff9700 1 -- 172.21.15.49:0/2287343302 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd4a004eec0 con 0x7fd4b0104e60 2023-07-20T17:42:33.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.431+0000 7fd4aeffd700 1 --2- 172.21.15.49:0/2287343302 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd4980386c0 0x7fd49803ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:33.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.432+0000 7fd4aeffd700 1 --2- 172.21.15.49:0/2287343302 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd4980386c0 0x7fd49803ab80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4006fd0 tx=0x7fd4a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:33.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.436+0000 7fd4acff9700 1 -- 172.21.15.49:0/2287343302 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd4a0027030 con 0x7fd4b0104e60 2023-07-20T17:42:33.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.792+0000 7fd4b5e43700 1 -- 172.21.15.49:0/2287343302 --> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4b0060bb0 con 0x7fd4980386c0 2023-07-20T17:42:33.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.793+0000 7fd4acff9700 1 -- 172.21.15.49:0/2287343302 <== mgr.14120 v2:172.21.15.49:6800/752420140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fd4b0060bb0 con 0x7fd4980386c0 2023-07-20T17:42:33.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.795+0000 7fd4b5e43700 1 -- 172.21.15.49:0/2287343302 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd4980386c0 msgr2=0x7fd49803ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:33.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.795+0000 7fd4b5e43700 1 --2- 172.21.15.49:0/2287343302 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd4980386c0 0x7fd49803ab80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4006fd0 tx=0x7fd4a4006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:33.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.795+0000 7fd4b5e43700 1 -- 172.21.15.49:0/2287343302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 msgr2=0x7fd4b01979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:33.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.795+0000 7fd4b5e43700 1 --2- 172.21.15.49:0/2287343302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b01979f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000d650 tx=0x7fd4a0003df0 comp rx=0 tx=0).stop 2023-07-20T17:42:33.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.795+0000 7fd4b5e43700 1 -- 172.21.15.49:0/2287343302 shutdown_connections 2023-07-20T17:42:33.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.795+0000 7fd4b5e43700 1 --2- 172.21.15.49:0/2287343302 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd4980386c0 0x7fd49803ab80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:33.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.795+0000 7fd4b5e43700 1 --2- 172.21.15.49:0/2287343302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b0104e60 0x7fd4b01979f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:33.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.795+0000 7fd4b5e43700 1 -- 172.21.15.49:0/2287343302 >> 172.21.15.49:0/2287343302 conn(0x7fd4b0100400 msgr2=0x7fd4b018e820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:33.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.795+0000 7fd4b5e43700 1 -- 172.21.15.49:0/2287343302 shutdown_connections 2023-07-20T17:42:33.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:32.795+0000 7fd4b5e43700 1 -- 172.21.15.49:0/2287343302 wait complete. 2023-07-20T17:42:33.217 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-20T17:42:33.217 INFO:teuthology.orchestra.run.smithi049.stdout:Adding key to root@localhost authorized_keys... 2023-07-20T17:42:33.217 INFO:teuthology.orchestra.run.smithi049.stdout:Adding host smithi049... 2023-07-20T17:42:33.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:33 smithi049 ceph-mon[96530]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:42:34.592 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:34 smithi049 ceph-mon[96530]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi049", "addr": "172.21.15.49", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:42:35.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:35 smithi049 ceph-mon[96530]: Deploying cephadm binary to smithi049 2023-07-20T17:42:37.261 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Added host 'smithi049' with addr '172.21.15.49' 2023-07-20T17:42:37.261 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.676+0000 7f65cb612700 1 Processor -- start 2023-07-20T17:42:37.262 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.676+0000 7f65cb612700 1 -- start start 2023-07-20T17:42:37.262 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.676+0000 7f65cb612700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4108e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:37.262 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.676+0000 7f65cb612700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c4079e70 con 0x7f65c4106a10 2023-07-20T17:42:37.262 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.676+0000 7f65c93ae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4108e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:37.262 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.676+0000 7f65c93ae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4108e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35300/0 (socket says 172.21.15.49:35300) 2023-07-20T17:42:37.262 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.676+0000 7f65c93ae700 1 -- 172.21.15.49:0/750987765 learned_addr learned my addr 172.21.15.49:0/750987765 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:37.262 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.677+0000 7f65c93ae700 1 -- 172.21.15.49:0/750987765 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c4109340 con 0x7f65c4106a10 2023-07-20T17:42:37.262 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.677+0000 7f65c93ae700 1 --2- 172.21.15.49:0/750987765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4108e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f65c00098a0 tx=0x7f65c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafa3c41792dac01 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:37.263 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.677+0000 7f65bbfff700 1 -- 172.21.15.49:0/750987765 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65c0004160 con 0x7f65c4106a10 2023-07-20T17:42:37.263 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.677+0000 7f65bbfff700 1 -- 172.21.15.49:0/750987765 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65c00042c0 con 0x7f65c4106a10 2023-07-20T17:42:37.263 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.677+0000 7f65bbfff700 1 -- 172.21.15.49:0/750987765 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65c0011a00 con 0x7f65c4106a10 2023-07-20T17:42:37.263 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.677+0000 7f65cb612700 1 -- 172.21.15.49:0/750987765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 msgr2=0x7f65c4108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:37.263 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.677+0000 7f65cb612700 1 --2- 172.21.15.49:0/750987765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4108e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f65c00098a0 tx=0x7f65c0009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:37.263 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.678+0000 7f65cb612700 1 -- 172.21.15.49:0/750987765 shutdown_connections 2023-07-20T17:42:37.263 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.678+0000 7f65cb612700 1 --2- 172.21.15.49:0/750987765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4108e00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:37.264 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.678+0000 7f65cb612700 1 -- 172.21.15.49:0/750987765 >> 172.21.15.49:0/750987765 conn(0x7f65c4100400 msgr2=0x7f65c4102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:37.264 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.678+0000 7f65cb612700 1 -- 172.21.15.49:0/750987765 shutdown_connections 2023-07-20T17:42:37.264 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.678+0000 7f65cb612700 1 -- 172.21.15.49:0/750987765 wait complete. 2023-07-20T17:42:37.264 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.678+0000 7f65cb612700 1 Processor -- start 2023-07-20T17:42:37.264 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.679+0000 7f65cb612700 1 -- start start 2023-07-20T17:42:37.264 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.679+0000 7f65cb612700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:37.264 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.679+0000 7f65cb612700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c419a170 con 0x7f65c4106a10 2023-07-20T17:42:37.264 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.679+0000 7f65c93ae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:37.265 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.679+0000 7f65c93ae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35302/0 (socket says 172.21.15.49:35302) 2023-07-20T17:42:37.265 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.679+0000 7f65c93ae700 1 -- 172.21.15.49:0/3492113907 learned_addr learned my addr 172.21.15.49:0/3492113907 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:37.265 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.679+0000 7f65c93ae700 1 -- 172.21.15.49:0/3492113907 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c0009550 con 0x7f65c4106a10 2023-07-20T17:42:37.265 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.679+0000 7f65c93ae700 1 --2- 172.21.15.49:0/3492113907 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4199c30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65c0009fb0 tx=0x7f65c0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:37.265 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.680+0000 7f65ba7fc700 1 -- 172.21.15.49:0/3492113907 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65c0003a40 con 0x7f65c4106a10 2023-07-20T17:42:37.265 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.680+0000 7f65cb612700 1 -- 172.21.15.49:0/3492113907 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c419a370 con 0x7f65c4106a10 2023-07-20T17:42:37.265 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.680+0000 7f65ba7fc700 1 -- 172.21.15.49:0/3492113907 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65c0003ba0 con 0x7f65c4106a10 2023-07-20T17:42:37.265 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.680+0000 7f65ba7fc700 1 -- 172.21.15.49:0/3492113907 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65c001a750 con 0x7f65c4106a10 2023-07-20T17:42:37.266 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.680+0000 7f65cb612700 1 -- 172.21.15.49:0/3492113907 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c419a790 con 0x7f65c4106a10 2023-07-20T17:42:37.266 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.681+0000 7f65ba7fc700 1 -- 172.21.15.49:0/3492113907 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f65c0024b60 con 0x7f65c4106a10 2023-07-20T17:42:37.266 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.681+0000 7f65cb612700 1 -- 172.21.15.49:0/3492113907 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c410a0e0 con 0x7f65c4106a10 2023-07-20T17:42:37.266 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.681+0000 7f65ba7fc700 1 --2- 172.21.15.49:0/3492113907 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f65b00386c0 0x7f65b003ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:37.266 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.681+0000 7f65ba7fc700 1 -- 172.21.15.49:0/3492113907 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f65c004c730 con 0x7f65c4106a10 2023-07-20T17:42:37.266 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.681+0000 7f65c8bad700 1 --2- 172.21.15.49:0/3492113907 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f65b00386c0 0x7f65b003ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:37.266 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.681+0000 7f65c8bad700 1 --2- 172.21.15.49:0/3492113907 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f65b00386c0 0x7f65b003ab80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f65b4006fd0 tx=0x7f65b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:37.267 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:33.686+0000 7f65ba7fc700 1 -- 172.21.15.49:0/3492113907 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f65c0018530 con 0x7f65c4106a10 2023-07-20T17:42:37.267 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:34.050+0000 7f65cb612700 1 -- 172.21.15.49:0/3492113907 --> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi049", "addr": "172.21.15.49", "target": ["mon-mgr", ""]}) v1 -- 0x7f65c41026d0 con 0x7f65b00386c0 2023-07-20T17:42:37.267 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:36.804+0000 7f65ba7fc700 1 -- 172.21.15.49:0/3492113907 <== mgr.14120 v2:172.21.15.49:6800/752420140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f65c41026d0 con 0x7f65b00386c0 2023-07-20T17:42:37.267 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:36.806+0000 7f65cb612700 1 -- 172.21.15.49:0/3492113907 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f65b00386c0 msgr2=0x7f65b003ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:37.267 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:36.806+0000 7f65cb612700 1 --2- 172.21.15.49:0/3492113907 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f65b00386c0 0x7f65b003ab80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f65b4006fd0 tx=0x7f65b4006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:37.267 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:36.806+0000 7f65cb612700 1 -- 172.21.15.49:0/3492113907 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 msgr2=0x7f65c4199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:37.267 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:36.806+0000 7f65cb612700 1 --2- 172.21.15.49:0/3492113907 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4199c30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65c0009fb0 tx=0x7f65c0011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:37.267 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:36.806+0000 7f65cb612700 1 -- 172.21.15.49:0/3492113907 shutdown_connections 2023-07-20T17:42:37.268 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:36.806+0000 7f65cb612700 1 --2- 172.21.15.49:0/3492113907 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f65b00386c0 0x7f65b003ab80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:37.268 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:36.806+0000 7f65cb612700 1 --2- 172.21.15.49:0/3492113907 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65c4106a10 0x7f65c4199c30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:37.268 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:36.807+0000 7f65cb612700 1 -- 172.21.15.49:0/3492113907 >> 172.21.15.49:0/3492113907 conn(0x7f65c4100400 msgr2=0x7f65c4101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:37.268 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:36.807+0000 7f65cb612700 1 -- 172.21.15.49:0/3492113907 shutdown_connections 2023-07-20T17:42:37.268 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:36.807+0000 7f65cb612700 1 -- 172.21.15.49:0/3492113907 wait complete. 2023-07-20T17:42:37.268 INFO:teuthology.orchestra.run.smithi049.stdout:Deploying unmanaged mon service... 2023-07-20T17:42:38.177 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:37 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:38.177 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:37 smithi049 ceph-mon[96530]: Added host smithi049 2023-07-20T17:42:38.177 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:37 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:42:38.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-20T17:42:38.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.734+0000 7fd9e1055700 1 Processor -- start 2023-07-20T17:42:38.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.734+0000 7fd9e1055700 1 -- start start 2023-07-20T17:42:38.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.734+0000 7fd9e1055700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc108e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:38.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.734+0000 7fd9e1055700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc079e70 con 0x7fd9dc106a10 2023-07-20T17:42:38.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.735+0000 7fd9dad9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc108e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:38.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.735+0000 7fd9dad9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc108e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35336/0 (socket says 172.21.15.49:35336) 2023-07-20T17:42:38.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.735+0000 7fd9dad9d700 1 -- 172.21.15.49:0/2152303450 learned_addr learned my addr 172.21.15.49:0/2152303450 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:38.910 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.735+0000 7fd9dad9d700 1 -- 172.21.15.49:0/2152303450 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc109340 con 0x7fd9dc106a10 2023-07-20T17:42:38.910 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.735+0000 7fd9dad9d700 1 --2- 172.21.15.49:0/2152303450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc108e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0098a0 tx=0x7fd9cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ae439c1cfa1dff server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:38.910 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.735+0000 7fd9d9d9b700 1 -- 172.21.15.49:0/2152303450 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9cc004160 con 0x7fd9dc106a10 2023-07-20T17:42:38.910 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.735+0000 7fd9d9d9b700 1 -- 172.21.15.49:0/2152303450 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9cc0042c0 con 0x7fd9dc106a10 2023-07-20T17:42:38.910 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.735+0000 7fd9d9d9b700 1 -- 172.21.15.49:0/2152303450 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9cc011a00 con 0x7fd9dc106a10 2023-07-20T17:42:38.910 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.736+0000 7fd9e1055700 1 -- 172.21.15.49:0/2152303450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 msgr2=0x7fd9dc108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:38.910 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.736+0000 7fd9e1055700 1 --2- 172.21.15.49:0/2152303450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc108e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0098a0 tx=0x7fd9cc009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:38.911 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.736+0000 7fd9e1055700 1 -- 172.21.15.49:0/2152303450 shutdown_connections 2023-07-20T17:42:38.911 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.736+0000 7fd9e1055700 1 --2- 172.21.15.49:0/2152303450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc108e00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:38.911 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.736+0000 7fd9e1055700 1 -- 172.21.15.49:0/2152303450 >> 172.21.15.49:0/2152303450 conn(0x7fd9dc100400 msgr2=0x7fd9dc102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:38.911 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.736+0000 7fd9e1055700 1 -- 172.21.15.49:0/2152303450 shutdown_connections 2023-07-20T17:42:38.911 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.736+0000 7fd9e1055700 1 -- 172.21.15.49:0/2152303450 wait complete. 2023-07-20T17:42:38.911 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.737+0000 7fd9e1055700 1 Processor -- start 2023-07-20T17:42:38.911 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.737+0000 7fd9e1055700 1 -- start start 2023-07-20T17:42:38.911 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.737+0000 7fd9e1055700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:38.912 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.737+0000 7fd9e1055700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc19a170 con 0x7fd9dc106a10 2023-07-20T17:42:38.912 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.737+0000 7fd9dad9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:38.912 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.737+0000 7fd9dad9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35338/0 (socket says 172.21.15.49:35338) 2023-07-20T17:42:38.912 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.737+0000 7fd9dad9d700 1 -- 172.21.15.49:0/2443132727 learned_addr learned my addr 172.21.15.49:0/2443132727 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:38.912 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.738+0000 7fd9dad9d700 1 -- 172.21.15.49:0/2443132727 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9cc009550 con 0x7fd9dc106a10 2023-07-20T17:42:38.912 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.738+0000 7fd9dad9d700 1 --2- 172.21.15.49:0/2443132727 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc199c30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009fb0 tx=0x7fd9cc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:38.912 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.738+0000 7fd9c3fff700 1 -- 172.21.15.49:0/2443132727 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9cc003a40 con 0x7fd9dc106a10 2023-07-20T17:42:38.913 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.738+0000 7fd9e1055700 1 -- 172.21.15.49:0/2443132727 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9dc19a370 con 0x7fd9dc106a10 2023-07-20T17:42:38.913 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.738+0000 7fd9c3fff700 1 -- 172.21.15.49:0/2443132727 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9cc003ba0 con 0x7fd9dc106a10 2023-07-20T17:42:38.913 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.738+0000 7fd9c3fff700 1 -- 172.21.15.49:0/2443132727 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9cc01a750 con 0x7fd9dc106a10 2023-07-20T17:42:38.913 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.738+0000 7fd9e1055700 1 -- 172.21.15.49:0/2443132727 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9dc19a790 con 0x7fd9dc106a10 2023-07-20T17:42:38.913 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.739+0000 7fd9c3fff700 1 -- 172.21.15.49:0/2443132727 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fd9cc01a9d0 con 0x7fd9dc106a10 2023-07-20T17:42:38.913 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.739+0000 7fd9e1055700 1 -- 172.21.15.49:0/2443132727 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9dc10a0e0 con 0x7fd9dc106a10 2023-07-20T17:42:38.913 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.739+0000 7fd9c3fff700 1 --2- 172.21.15.49:0/2443132727 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd9c40386c0 0x7fd9c403ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:38.914 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.739+0000 7fd9c3fff700 1 -- 172.21.15.49:0/2443132727 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd9cc04c740 con 0x7fd9dc106a10 2023-07-20T17:42:38.914 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.739+0000 7fd9da59c700 1 --2- 172.21.15.49:0/2443132727 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd9c40386c0 0x7fd9c403ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:38.914 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.740+0000 7fd9da59c700 1 --2- 172.21.15.49:0/2443132727 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd9c40386c0 0x7fd9c403ab80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0006fd0 tx=0x7fd9d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:38.914 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:37.744+0000 7fd9c3fff700 1 -- 172.21.15.49:0/2443132727 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd9cc018b80 con 0x7fd9dc106a10 2023-07-20T17:42:38.914 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.140+0000 7fd9e1055700 1 -- 172.21.15.49:0/2443132727 --> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd9dc1026d0 con 0x7fd9c40386c0 2023-07-20T17:42:38.914 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.146+0000 7fd9c3fff700 1 -- 172.21.15.49:0/2443132727 <== mgr.14120 v2:172.21.15.49:6800/752420140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd9dc1026d0 con 0x7fd9c40386c0 2023-07-20T17:42:38.914 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.149+0000 7fd9e1055700 1 -- 172.21.15.49:0/2443132727 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd9c40386c0 msgr2=0x7fd9c403ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:38.915 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.149+0000 7fd9e1055700 1 --2- 172.21.15.49:0/2443132727 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd9c40386c0 0x7fd9c403ab80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0006fd0 tx=0x7fd9d0006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:38.915 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.149+0000 7fd9e1055700 1 -- 172.21.15.49:0/2443132727 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 msgr2=0x7fd9dc199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:38.915 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.149+0000 7fd9e1055700 1 --2- 172.21.15.49:0/2443132727 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc199c30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009fb0 tx=0x7fd9cc011ad0 comp rx=0 tx=0).stop 2023-07-20T17:42:38.915 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.149+0000 7fd9e1055700 1 -- 172.21.15.49:0/2443132727 shutdown_connections 2023-07-20T17:42:38.915 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.149+0000 7fd9e1055700 1 --2- 172.21.15.49:0/2443132727 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fd9c40386c0 0x7fd9c403ab80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:38.915 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.149+0000 7fd9e1055700 1 --2- 172.21.15.49:0/2443132727 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9dc106a10 0x7fd9dc199c30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:38.915 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.149+0000 7fd9e1055700 1 -- 172.21.15.49:0/2443132727 >> 172.21.15.49:0/2443132727 conn(0x7fd9dc100400 msgr2=0x7fd9dc101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:38.915 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.150+0000 7fd9e1055700 1 -- 172.21.15.49:0/2443132727 shutdown_connections 2023-07-20T17:42:38.916 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:38.150+0000 7fd9e1055700 1 -- 172.21.15.49:0/2443132727 wait complete. 2023-07-20T17:42:38.916 INFO:teuthology.orchestra.run.smithi049.stdout:Deploying unmanaged mgr service... 2023-07-20T17:42:39.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:39 smithi049 ceph-mon[96530]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:42:39.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:39 smithi049 ceph-mon[96530]: Saving service mon spec with placement count:5 2023-07-20T17:42:39.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:39 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:40.487 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:40 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:40.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:40 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:40.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:40 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:40.512 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-20T17:42:40.512 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.656+0000 7f3a4657e700 1 Processor -- start 2023-07-20T17:42:40.513 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.656+0000 7f3a4657e700 1 -- start start 2023-07-20T17:42:40.513 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.656+0000 7f3a4657e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a40108e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:40.513 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.656+0000 7f3a4657e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a40079e70 con 0x7f3a40106a10 2023-07-20T17:42:40.513 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.656+0000 7f3a3ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a40108e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:40.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.657+0000 7f3a3ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a40108e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35342/0 (socket says 172.21.15.49:35342) 2023-07-20T17:42:40.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.657+0000 7f3a3ffff700 1 -- 172.21.15.49:0/467747728 learned_addr learned my addr 172.21.15.49:0/467747728 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:40.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.657+0000 7f3a3ffff700 1 -- 172.21.15.49:0/467747728 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a40109340 con 0x7f3a40106a10 2023-07-20T17:42:40.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.657+0000 7f3a3ffff700 1 --2- 172.21.15.49:0/467747728 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a40108e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000f960 tx=0x7f3a3000fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ebb6a9d29d2cf3 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:40.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.657+0000 7f3a3effd700 1 -- 172.21.15.49:0/467747728 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a30009b30 con 0x7f3a40106a10 2023-07-20T17:42:40.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.657+0000 7f3a3effd700 1 -- 172.21.15.49:0/467747728 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a300044f0 con 0x7f3a40106a10 2023-07-20T17:42:40.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.657+0000 7f3a3effd700 1 -- 172.21.15.49:0/467747728 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a30003db0 con 0x7f3a40106a10 2023-07-20T17:42:40.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.658+0000 7f3a4657e700 1 -- 172.21.15.49:0/467747728 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 msgr2=0x7f3a40108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:40.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.658+0000 7f3a4657e700 1 --2- 172.21.15.49:0/467747728 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a40108e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000f960 tx=0x7f3a3000fc70 comp rx=0 tx=0).stop 2023-07-20T17:42:40.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.658+0000 7f3a4657e700 1 -- 172.21.15.49:0/467747728 shutdown_connections 2023-07-20T17:42:40.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.658+0000 7f3a4657e700 1 --2- 172.21.15.49:0/467747728 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a40108e00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:40.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.658+0000 7f3a4657e700 1 -- 172.21.15.49:0/467747728 >> 172.21.15.49:0/467747728 conn(0x7f3a40100400 msgr2=0x7f3a40102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:40.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.658+0000 7f3a4657e700 1 -- 172.21.15.49:0/467747728 shutdown_connections 2023-07-20T17:42:40.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.658+0000 7f3a4657e700 1 -- 172.21.15.49:0/467747728 wait complete. 2023-07-20T17:42:40.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.659+0000 7f3a4657e700 1 Processor -- start 2023-07-20T17:42:40.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.659+0000 7f3a4657e700 1 -- start start 2023-07-20T17:42:40.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.659+0000 7f3a4657e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a401979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:40.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.659+0000 7f3a4657e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a40197f00 con 0x7f3a40106a10 2023-07-20T17:42:40.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.659+0000 7f3a3ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a401979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:40.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.659+0000 7f3a3ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a401979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35344/0 (socket says 172.21.15.49:35344) 2023-07-20T17:42:40.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.659+0000 7f3a3ffff700 1 -- 172.21.15.49:0/4173437245 learned_addr learned my addr 172.21.15.49:0/4173437245 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:40.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.659+0000 7f3a3ffff700 1 -- 172.21.15.49:0/4173437245 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3000f610 con 0x7f3a40106a10 2023-07-20T17:42:40.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.660+0000 7f3a3ffff700 1 --2- 172.21.15.49:0/4173437245 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a401979c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3a30000c00 tx=0x7f3a300047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:40.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.660+0000 7f3a3d7fa700 1 -- 172.21.15.49:0/4173437245 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a30017700 con 0x7f3a40106a10 2023-07-20T17:42:40.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.660+0000 7f3a3d7fa700 1 -- 172.21.15.49:0/4173437245 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a30017d20 con 0x7f3a40106a10 2023-07-20T17:42:40.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.660+0000 7f3a4657e700 1 -- 172.21.15.49:0/4173437245 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a40198100 con 0x7f3a40106a10 2023-07-20T17:42:40.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.660+0000 7f3a3d7fa700 1 -- 172.21.15.49:0/4173437245 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a30020b10 con 0x7f3a40106a10 2023-07-20T17:42:40.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.660+0000 7f3a4657e700 1 -- 172.21.15.49:0/4173437245 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a401984c0 con 0x7f3a40106a10 2023-07-20T17:42:40.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.661+0000 7f3a4657e700 1 -- 172.21.15.49:0/4173437245 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a40191990 con 0x7f3a40106a10 2023-07-20T17:42:40.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.661+0000 7f3a3d7fa700 1 -- 172.21.15.49:0/4173437245 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f3a3001e070 con 0x7f3a40106a10 2023-07-20T17:42:40.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.661+0000 7f3a3d7fa700 1 --2- 172.21.15.49:0/4173437245 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3a28038710 0x7f3a2803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:40.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.661+0000 7f3a3d7fa700 1 -- 172.21.15.49:0/4173437245 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3a30050220 con 0x7f3a40106a10 2023-07-20T17:42:40.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.662+0000 7f3a3f7fe700 1 --2- 172.21.15.49:0/4173437245 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3a28038710 0x7f3a2803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:40.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.662+0000 7f3a3f7fe700 1 --2- 172.21.15.49:0/4173437245 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3a28038710 0x7f3a2803abd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3a34006fd0 tx=0x7f3a34006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:40.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:39.670+0000 7f3a3d7fa700 1 -- 172.21.15.49:0/4173437245 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3a30016590 con 0x7f3a40106a10 2023-07-20T17:42:40.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.065+0000 7f3a4657e700 1 -- 172.21.15.49:0/4173437245 --> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3a40060bb0 con 0x7f3a28038710 2023-07-20T17:42:40.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.070+0000 7f3a3d7fa700 1 -- 172.21.15.49:0/4173437245 <== mgr.14120 v2:172.21.15.49:6800/752420140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3a40060bb0 con 0x7f3a28038710 2023-07-20T17:42:40.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.072+0000 7f3a4657e700 1 -- 172.21.15.49:0/4173437245 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3a28038710 msgr2=0x7f3a2803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:40.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.072+0000 7f3a4657e700 1 --2- 172.21.15.49:0/4173437245 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3a28038710 0x7f3a2803abd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3a34006fd0 tx=0x7f3a34006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:40.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.072+0000 7f3a4657e700 1 -- 172.21.15.49:0/4173437245 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 msgr2=0x7f3a401979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:40.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.072+0000 7f3a4657e700 1 --2- 172.21.15.49:0/4173437245 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a401979c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3a30000c00 tx=0x7f3a300047c0 comp rx=0 tx=0).stop 2023-07-20T17:42:40.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.072+0000 7f3a4657e700 1 -- 172.21.15.49:0/4173437245 shutdown_connections 2023-07-20T17:42:40.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.072+0000 7f3a4657e700 1 --2- 172.21.15.49:0/4173437245 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3a28038710 0x7f3a2803abd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:40.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.072+0000 7f3a4657e700 1 --2- 172.21.15.49:0/4173437245 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a40106a10 0x7f3a401979c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:40.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.072+0000 7f3a4657e700 1 -- 172.21.15.49:0/4173437245 >> 172.21.15.49:0/4173437245 conn(0x7f3a40100400 msgr2=0x7f3a401010c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:40.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.072+0000 7f3a4657e700 1 -- 172.21.15.49:0/4173437245 shutdown_connections 2023-07-20T17:42:40.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:40.072+0000 7f3a4657e700 1 -- 172.21.15.49:0/4173437245 wait complete. 2023-07-20T17:42:41.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:41 smithi049 ceph-mon[96530]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:42:41.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:41 smithi049 ceph-mon[96530]: Saving service mgr spec with placement count:2 2023-07-20T17:42:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.027+0000 7f470e062700 1 Processor -- start 2023-07-20T17:42:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.027+0000 7f470e062700 1 -- start start 2023-07-20T17:42:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.027+0000 7f470e062700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708105270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.027+0000 7f470e062700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47081057b0 con 0x7f4708104e50 2023-07-20T17:42:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.027+0000 7f47077fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708105270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.028+0000 7f47077fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708105270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35348/0 (socket says 172.21.15.49:35348) 2023-07-20T17:42:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.028+0000 7f47077fe700 1 -- 172.21.15.49:0/2157902521 learned_addr learned my addr 172.21.15.49:0/2157902521 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.028+0000 7f47077fe700 1 -- 172.21.15.49:0/2157902521 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47081058f0 con 0x7f4708104e50 2023-07-20T17:42:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.028+0000 7f47077fe700 1 --2- 172.21.15.49:0/2157902521 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708105270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f46f80098a0 tx=0x7f46f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ae3f2b8eb542ce server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.028+0000 7f47067fc700 1 -- 172.21.15.49:0/2157902521 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46f8004160 con 0x7f4708104e50 2023-07-20T17:42:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.028+0000 7f47067fc700 1 -- 172.21.15.49:0/2157902521 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46f80042c0 con 0x7f4708104e50 2023-07-20T17:42:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.028+0000 7f47067fc700 1 -- 172.21.15.49:0/2157902521 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46f80119b0 con 0x7f4708104e50 2023-07-20T17:42:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.029+0000 7f470e062700 1 -- 172.21.15.49:0/2157902521 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 msgr2=0x7f4708105270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.029+0000 7f470e062700 1 --2- 172.21.15.49:0/2157902521 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708105270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f46f80098a0 tx=0x7f46f8009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.029+0000 7f470e062700 1 -- 172.21.15.49:0/2157902521 shutdown_connections 2023-07-20T17:42:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.029+0000 7f470e062700 1 --2- 172.21.15.49:0/2157902521 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708105270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.029+0000 7f470e062700 1 -- 172.21.15.49:0/2157902521 >> 172.21.15.49:0/2157902521 conn(0x7f4708100430 msgr2=0x7f4708102850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:41.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.029+0000 7f470e062700 1 -- 172.21.15.49:0/2157902521 shutdown_connections 2023-07-20T17:42:41.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.029+0000 7f470e062700 1 -- 172.21.15.49:0/2157902521 wait complete. 2023-07-20T17:42:41.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.029+0000 7f470e062700 1 Processor -- start 2023-07-20T17:42:41.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.030+0000 7f470e062700 1 -- start start 2023-07-20T17:42:41.766 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.030+0000 7f470e062700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708079b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:41.766 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.030+0000 7f470e062700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470819a700 con 0x7f4708104e50 2023-07-20T17:42:41.766 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.030+0000 7f47077fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708079b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:41.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.030+0000 7f47077fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708079b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35350/0 (socket says 172.21.15.49:35350) 2023-07-20T17:42:41.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.030+0000 7f47077fe700 1 -- 172.21.15.49:0/1400999023 learned_addr learned my addr 172.21.15.49:0/1400999023 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:41.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.030+0000 7f47077fe700 1 -- 172.21.15.49:0/1400999023 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f8009550 con 0x7f4708104e50 2023-07-20T17:42:41.767 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.030+0000 7f47077fe700 1 --2- 172.21.15.49:0/1400999023 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708079b30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f46f8011980 tx=0x7f46f8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:41.768 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.030+0000 7f4704ff9700 1 -- 172.21.15.49:0/1400999023 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46f8003a60 con 0x7f4708104e50 2023-07-20T17:42:41.768 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.031+0000 7f4704ff9700 1 -- 172.21.15.49:0/1400999023 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46f8003bc0 con 0x7f4708104e50 2023-07-20T17:42:41.768 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.031+0000 7f470e062700 1 -- 172.21.15.49:0/1400999023 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470819ed10 con 0x7f4708104e50 2023-07-20T17:42:41.769 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.031+0000 7f4704ff9700 1 -- 172.21.15.49:0/1400999023 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46f801a670 con 0x7f4708104e50 2023-07-20T17:42:41.769 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.031+0000 7f470e062700 1 -- 172.21.15.49:0/1400999023 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470819f070 con 0x7f4708104e50 2023-07-20T17:42:41.769 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.031+0000 7f4704ff9700 1 -- 172.21.15.49:0/1400999023 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f46f801a8e0 con 0x7f4708104e50 2023-07-20T17:42:41.770 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.032+0000 7f4704ff9700 1 --2- 172.21.15.49:0/1400999023 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f46f4038760 0x7f46f403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.032+0000 7f4704ff9700 1 -- 172.21.15.49:0/1400999023 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f46f804c940 con 0x7f4708104e50 2023-07-20T17:42:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.032+0000 7f4706ffd700 1 --2- 172.21.15.49:0/1400999023 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f46f4038760 0x7f46f403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.032+0000 7f470e062700 1 -- 172.21.15.49:0/1400999023 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4708061da0 con 0x7f4708104e50 2023-07-20T17:42:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.032+0000 7f4706ffd700 1 --2- 172.21.15.49:0/1400999023 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f46f4038760 0x7f46f403ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f46f0006fd0 tx=0x7f46f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.037+0000 7f4704ff9700 1 -- 172.21.15.49:0/1400999023 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f46f801ab90 con 0x7f4708104e50 2023-07-20T17:42:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.406+0000 7f470e062700 1 -- 172.21.15.49:0/1400999023 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f4708109240 con 0x7f4708104e50 2023-07-20T17:42:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.409+0000 7f4704ff9700 1 -- 172.21.15.49:0/1400999023 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f46f8018350 con 0x7f4708104e50 2023-07-20T17:42:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.412+0000 7f470e062700 1 -- 172.21.15.49:0/1400999023 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f46f4038760 msgr2=0x7f46f403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.412+0000 7f470e062700 1 --2- 172.21.15.49:0/1400999023 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f46f4038760 0x7f46f403ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f46f0006fd0 tx=0x7f46f0006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.412+0000 7f470e062700 1 -- 172.21.15.49:0/1400999023 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 msgr2=0x7f4708079b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.412+0000 7f470e062700 1 --2- 172.21.15.49:0/1400999023 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708079b30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f46f8011980 tx=0x7f46f8011a60 comp rx=0 tx=0).stop 2023-07-20T17:42:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.412+0000 7f470e062700 1 -- 172.21.15.49:0/1400999023 shutdown_connections 2023-07-20T17:42:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.412+0000 7f470e062700 1 --2- 172.21.15.49:0/1400999023 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f46f4038760 0x7f46f403ac20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.412+0000 7f470e062700 1 --2- 172.21.15.49:0/1400999023 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4708104e50 0x7f4708079b30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.412+0000 7f470e062700 1 -- 172.21.15.49:0/1400999023 >> 172.21.15.49:0/1400999023 conn(0x7f4708100430 msgr2=0x7f47081011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.413+0000 7f470e062700 1 -- 172.21.15.49:0/1400999023 shutdown_connections 2023-07-20T17:42:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:41.413+0000 7f470e062700 1 -- 172.21.15.49:0/1400999023 wait complete. 2023-07-20T17:42:42.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:42 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:42.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:42 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1400999023' entity='client.admin' 2023-07-20T17:42:42.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:42 smithi049 ceph-mon[96530]: from='mgr.14120 172.21.15.49:0/2299071805' entity='mgr.x' 2023-07-20T17:42:43.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.300+0000 7f5663520700 1 Processor -- start 2023-07-20T17:42:43.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.300+0000 7f5663520700 1 -- start start 2023-07-20T17:42:43.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.300+0000 7f5663520700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c106340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:43.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.300+0000 7f5663520700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c106880 con 0x7f565c105f00 2023-07-20T17:42:43.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.301+0000 7f56612bc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c106340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:43.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.301+0000 7f56612bc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c106340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35354/0 (socket says 172.21.15.49:35354) 2023-07-20T17:42:43.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.301+0000 7f56612bc700 1 -- 172.21.15.49:0/4230525664 learned_addr learned my addr 172.21.15.49:0/4230525664 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:43.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.301+0000 7f56612bc700 1 -- 172.21.15.49:0/4230525664 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c1069c0 con 0x7f565c105f00 2023-07-20T17:42:43.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.301+0000 7f56612bc700 1 --2- 172.21.15.49:0/4230525664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c106340 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f56580098a0 tx=0x7f5658009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c47bef6237c812 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:43.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.302+0000 7f5653fff700 1 -- 172.21.15.49:0/4230525664 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5658004160 con 0x7f565c105f00 2023-07-20T17:42:43.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.302+0000 7f5653fff700 1 -- 172.21.15.49:0/4230525664 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56580042c0 con 0x7f565c105f00 2023-07-20T17:42:43.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.302+0000 7f5653fff700 1 -- 172.21.15.49:0/4230525664 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56580119b0 con 0x7f565c105f00 2023-07-20T17:42:43.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.302+0000 7f5663520700 1 -- 172.21.15.49:0/4230525664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 msgr2=0x7f565c106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:43.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.302+0000 7f5663520700 1 --2- 172.21.15.49:0/4230525664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c106340 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f56580098a0 tx=0x7f5658009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:43.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.302+0000 7f5663520700 1 -- 172.21.15.49:0/4230525664 shutdown_connections 2023-07-20T17:42:43.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.302+0000 7f5663520700 1 --2- 172.21.15.49:0/4230525664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c106340 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:43.107 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.302+0000 7f5663520700 1 -- 172.21.15.49:0/4230525664 >> 172.21.15.49:0/4230525664 conn(0x7f565c101550 msgr2=0x7f565c103970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:43.107 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.303+0000 7f5663520700 1 -- 172.21.15.49:0/4230525664 shutdown_connections 2023-07-20T17:42:43.107 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.303+0000 7f5663520700 1 -- 172.21.15.49:0/4230525664 wait complete. 2023-07-20T17:42:43.107 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.303+0000 7f5663520700 1 Processor -- start 2023-07-20T17:42:43.108 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.303+0000 7f5663520700 1 -- start start 2023-07-20T17:42:43.108 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.303+0000 7f5663520700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c07bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:43.108 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.303+0000 7f5663520700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c07c190 con 0x7f565c105f00 2023-07-20T17:42:43.108 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.304+0000 7f56612bc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c07bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:43.108 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.304+0000 7f56612bc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c07bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35356/0 (socket says 172.21.15.49:35356) 2023-07-20T17:42:43.108 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.304+0000 7f56612bc700 1 -- 172.21.15.49:0/2339508065 learned_addr learned my addr 172.21.15.49:0/2339508065 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:43.108 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.304+0000 7f56612bc700 1 -- 172.21.15.49:0/2339508065 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5658009550 con 0x7f565c105f00 2023-07-20T17:42:43.109 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.304+0000 7f56612bc700 1 --2- 172.21.15.49:0/2339508065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c07bc50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5658003e60 tx=0x7f5658011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:43.109 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.304+0000 7f56527fc700 1 -- 172.21.15.49:0/2339508065 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5658003970 con 0x7f565c105f00 2023-07-20T17:42:43.109 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.304+0000 7f56527fc700 1 -- 172.21.15.49:0/2339508065 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5658003ad0 con 0x7f565c105f00 2023-07-20T17:42:43.109 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.304+0000 7f5663520700 1 -- 172.21.15.49:0/2339508065 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c07a300 con 0x7f565c105f00 2023-07-20T17:42:43.109 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.304+0000 7f56527fc700 1 -- 172.21.15.49:0/2339508065 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5658019590 con 0x7f565c105f00 2023-07-20T17:42:43.109 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.304+0000 7f5663520700 1 -- 172.21.15.49:0/2339508065 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c07a780 con 0x7f565c105f00 2023-07-20T17:42:43.109 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.305+0000 7f56527fc700 1 -- 172.21.15.49:0/2339508065 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f5658019800 con 0x7f565c105f00 2023-07-20T17:42:43.110 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.305+0000 7f56527fc700 1 --2- 172.21.15.49:0/2339508065 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f56480386f0 0x7f564803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:43.110 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.305+0000 7f56527fc700 1 -- 172.21.15.49:0/2339508065 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f565804c840 con 0x7f565c105f00 2023-07-20T17:42:43.110 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.305+0000 7f5660abb700 1 --2- 172.21.15.49:0/2339508065 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f56480386f0 0x7f564803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:43.110 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.306+0000 7f5660abb700 1 --2- 172.21.15.49:0/2339508065 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f56480386f0 0x7f564803abb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f564c006fd0 tx=0x7f564c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:43.110 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.306+0000 7f5663520700 1 -- 172.21.15.49:0/2339508065 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c061da0 con 0x7f565c105f00 2023-07-20T17:42:43.110 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.312+0000 7f56527fc700 1 -- 172.21.15.49:0/2339508065 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5658019b00 con 0x7f565c105f00 2023-07-20T17:42:43.110 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.682+0000 7f5663520700 1 -- 172.21.15.49:0/2339508065 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f565c107df0 con 0x7f565c105f00 2023-07-20T17:42:43.110 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.685+0000 7f56527fc700 1 -- 172.21.15.49:0/2339508065 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f565c107df0 con 0x7f565c105f00 2023-07-20T17:42:43.111 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.688+0000 7f5663520700 1 -- 172.21.15.49:0/2339508065 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f56480386f0 msgr2=0x7f564803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:43.111 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.688+0000 7f5663520700 1 --2- 172.21.15.49:0/2339508065 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f56480386f0 0x7f564803abb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f564c006fd0 tx=0x7f564c006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:43.111 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.688+0000 7f5663520700 1 -- 172.21.15.49:0/2339508065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 msgr2=0x7f565c07bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:43.111 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.688+0000 7f5663520700 1 --2- 172.21.15.49:0/2339508065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c07bc50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5658003e60 tx=0x7f5658011aa0 comp rx=0 tx=0).stop 2023-07-20T17:42:43.111 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.688+0000 7f5663520700 1 -- 172.21.15.49:0/2339508065 shutdown_connections 2023-07-20T17:42:43.111 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.688+0000 7f5663520700 1 --2- 172.21.15.49:0/2339508065 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f56480386f0 0x7f564803abb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:43.111 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.688+0000 7f5663520700 1 --2- 172.21.15.49:0/2339508065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f565c105f00 0x7f565c07bc50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:43.112 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.688+0000 7f5663520700 1 -- 172.21.15.49:0/2339508065 >> 172.21.15.49:0/2339508065 conn(0x7f565c101550 msgr2=0x7f565c102330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:43.112 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.688+0000 7f5663520700 1 -- 172.21.15.49:0/2339508065 shutdown_connections 2023-07-20T17:42:43.112 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:42.688+0000 7f5663520700 1 -- 172.21.15.49:0/2339508065 wait complete. 2023-07-20T17:42:43.112 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling the dashboard module... 2023-07-20T17:42:43.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:43 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2339508065' entity='client.admin' 2023-07-20T17:42:44.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:44 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1280399130' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-20T17:42:44.989 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:44 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: ignoring --setuser ceph since I am not root 2023-07-20T17:42:44.989 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:44 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: ignoring --setgroup ceph since I am not root 2023-07-20T17:42:44.989 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:44 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:44.893+0000 7f044c205200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T17:42:45.166 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.721+0000 7fc661695700 1 Processor -- start 2023-07-20T17:42:45.166 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.721+0000 7fc661695700 1 -- start start 2023-07-20T17:42:45.167 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.721+0000 7fc661695700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c106810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:45.167 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.721+0000 7fc661695700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c106de0 con 0x7fc65c1063f0 2023-07-20T17:42:45.168 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.721+0000 7fc65affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:45.168 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.721+0000 7fc65affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35360/0 (socket says 172.21.15.49:35360) 2023-07-20T17:42:45.168 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.721+0000 7fc65affd700 1 -- 172.21.15.49:0/310128006 learned_addr learned my addr 172.21.15.49:0/310128006 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:45.168 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.722+0000 7fc65affd700 1 -- 172.21.15.49:0/310128006 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c105050 con 0x7fc65c1063f0 2023-07-20T17:42:45.168 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.722+0000 7fc65affd700 1 --2- 172.21.15.49:0/310128006 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c106810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0098a0 tx=0x7fc64c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104875874916a206 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:45.169 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.722+0000 7fc659ffb700 1 -- 172.21.15.49:0/310128006 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc64c004160 con 0x7fc65c1063f0 2023-07-20T17:42:45.169 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.722+0000 7fc659ffb700 1 -- 172.21.15.49:0/310128006 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc64c0042c0 con 0x7fc65c1063f0 2023-07-20T17:42:45.169 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.722+0000 7fc659ffb700 1 -- 172.21.15.49:0/310128006 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc64c0119b0 con 0x7fc65c1063f0 2023-07-20T17:42:45.169 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.722+0000 7fc661695700 1 -- 172.21.15.49:0/310128006 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 msgr2=0x7fc65c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:45.169 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.722+0000 7fc661695700 1 --2- 172.21.15.49:0/310128006 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c106810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0098a0 tx=0x7fc64c009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:45.170 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.723+0000 7fc661695700 1 -- 172.21.15.49:0/310128006 shutdown_connections 2023-07-20T17:42:45.170 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.723+0000 7fc661695700 1 --2- 172.21.15.49:0/310128006 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c106810 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:45.170 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.723+0000 7fc661695700 1 -- 172.21.15.49:0/310128006 >> 172.21.15.49:0/310128006 conn(0x7fc65c0ffbc0 msgr2=0x7fc65c101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:45.170 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.723+0000 7fc661695700 1 -- 172.21.15.49:0/310128006 shutdown_connections 2023-07-20T17:42:45.170 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.723+0000 7fc661695700 1 -- 172.21.15.49:0/310128006 wait complete. 2023-07-20T17:42:45.170 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.723+0000 7fc661695700 1 Processor -- start 2023-07-20T17:42:45.170 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.723+0000 7fc661695700 1 -- start start 2023-07-20T17:42:45.170 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc661695700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c190020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:45.171 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc661695700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c190560 con 0x7fc65c1063f0 2023-07-20T17:42:45.171 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc65affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c190020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:45.171 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc65affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c190020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35362/0 (socket says 172.21.15.49:35362) 2023-07-20T17:42:45.171 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc65affd700 1 -- 172.21.15.49:0/1280399130 learned_addr learned my addr 172.21.15.49:0/1280399130 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:45.171 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc65affd700 1 -- 172.21.15.49:0/1280399130 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c009550 con 0x7fc65c1063f0 2023-07-20T17:42:45.171 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc65affd700 1 --2- 172.21.15.49:0/1280399130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c190020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc64c003e60 tx=0x7fc64c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:45.171 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc643fff700 1 -- 172.21.15.49:0/1280399130 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc64c003970 con 0x7fc65c1063f0 2023-07-20T17:42:45.172 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc643fff700 1 -- 172.21.15.49:0/1280399130 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc64c003ad0 con 0x7fc65c1063f0 2023-07-20T17:42:45.172 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc661695700 1 -- 172.21.15.49:0/1280399130 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65c18e670 con 0x7fc65c1063f0 2023-07-20T17:42:45.172 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc643fff700 1 -- 172.21.15.49:0/1280399130 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc64c0195e0 con 0x7fc65c1063f0 2023-07-20T17:42:45.172 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.724+0000 7fc661695700 1 -- 172.21.15.49:0/1280399130 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65c18ea90 con 0x7fc65c1063f0 2023-07-20T17:42:45.172 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.725+0000 7fc643fff700 1 -- 172.21.15.49:0/1280399130 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fc64c019870 con 0x7fc65c1063f0 2023-07-20T17:42:45.172 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.725+0000 7fc661695700 1 -- 172.21.15.49:0/1280399130 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65c109290 con 0x7fc65c1063f0 2023-07-20T17:42:45.172 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.726+0000 7fc643fff700 1 --2- 172.21.15.49:0/1280399130 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fc644038710 0x7fc64403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:45.172 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.726+0000 7fc643fff700 1 -- 172.21.15.49:0/1280399130 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc64c04c850 con 0x7fc65c1063f0 2023-07-20T17:42:45.173 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.726+0000 7fc65a7fc700 1 --2- 172.21.15.49:0/1280399130 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fc644038710 0x7fc64403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:45.173 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.726+0000 7fc65a7fc700 1 --2- 172.21.15.49:0/1280399130 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fc644038710 0x7fc64403abd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc650006fd0 tx=0x7fc650006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:45.173 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:43.731+0000 7fc643fff700 1 -- 172.21.15.49:0/1280399130 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc64c02d440 con 0x7fc65c1063f0 2023-07-20T17:42:45.173 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.204+0000 7fc661695700 1 -- 172.21.15.49:0/1280399130 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fc65c109480 con 0x7fc65c1063f0 2023-07-20T17:42:45.173 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.688+0000 7fc643fff700 1 -- 172.21.15.49:0/1280399130 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fc64c02d9c0 con 0x7fc65c1063f0 2023-07-20T17:42:45.173 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.690+0000 7fc643fff700 1 -- 172.21.15.49:0/1280399130 <== mon.0 v2:172.21.15.49:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fc64c01aa30 con 0x7fc65c1063f0 2023-07-20T17:42:45.173 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.692+0000 7fc661695700 1 -- 172.21.15.49:0/1280399130 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fc644038710 msgr2=0x7fc64403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:45.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.693+0000 7fc661695700 1 --2- 172.21.15.49:0/1280399130 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fc644038710 0x7fc64403abd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc650006fd0 tx=0x7fc650006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:45.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.693+0000 7fc661695700 1 -- 172.21.15.49:0/1280399130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 msgr2=0x7fc65c190020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:45.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.693+0000 7fc661695700 1 --2- 172.21.15.49:0/1280399130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c190020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc64c003e60 tx=0x7fc64c011aa0 comp rx=0 tx=0).stop 2023-07-20T17:42:45.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.693+0000 7fc661695700 1 -- 172.21.15.49:0/1280399130 shutdown_connections 2023-07-20T17:42:45.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.693+0000 7fc661695700 1 --2- 172.21.15.49:0/1280399130 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fc644038710 0x7fc64403abd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:45.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.693+0000 7fc661695700 1 --2- 172.21.15.49:0/1280399130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65c1063f0 0x7fc65c190020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:45.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.693+0000 7fc661695700 1 -- 172.21.15.49:0/1280399130 >> 172.21.15.49:0/1280399130 conn(0x7fc65c0ffbc0 msgr2=0x7fc65c1047d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:45.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.693+0000 7fc661695700 1 -- 172.21.15.49:0/1280399130 shutdown_connections 2023-07-20T17:42:45.175 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:44.693+0000 7fc661695700 1 -- 172.21.15.49:0/1280399130 wait complete. 2023-07-20T17:42:45.369 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:44 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:44.988+0000 7f044c205200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T17:42:45.369 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:45.098+0000 7f044c205200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T17:42:45.369 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:45.220+0000 7f044c205200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T17:42:45.687 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:45.433+0000 7f044c205200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T17:42:45.941 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:45.831+0000 7f044c205200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T17:42:45.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:45 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1280399130' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-20T17:42:45.942 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:45 smithi049 ceph-mon[96530]: mgrmap e9: x(active, since 18s) 2023-07-20T17:42:46.220 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:45.940+0000 7f044c205200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T17:42:46.221 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:46 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:46.099+0000 7f044c205200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T17:42:46.488 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:46 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:46.220+0000 7f044c205200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T17:42:46.513 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2023-07-20T17:42:46.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-20T17:42:46.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T17:42:46.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T17:42:46.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T17:42:46.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2023-07-20T17:42:46.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.649+0000 7fefc78e1700 1 Processor -- start 2023-07-20T17:42:46.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.649+0000 7fefc78e1700 1 -- start start 2023-07-20T17:42:46.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.650+0000 7fefc78e1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0106340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:46.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.650+0000 7fefc78e1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc0106880 con 0x7fefc0105f00 2023-07-20T17:42:46.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.650+0000 7fefc567d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0106340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:46.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.650+0000 7fefc567d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0106340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35376/0 (socket says 172.21.15.49:35376) 2023-07-20T17:42:46.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.650+0000 7fefc567d700 1 -- 172.21.15.49:0/2786814333 learned_addr learned my addr 172.21.15.49:0/2786814333 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:46.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.650+0000 7fefc567d700 1 -- 172.21.15.49:0/2786814333 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc01069c0 con 0x7fefc0105f00 2023-07-20T17:42:46.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.650+0000 7fefc567d700 1 --2- 172.21.15.49:0/2786814333 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0106340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0098a0 tx=0x7fefbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e45192cfbddab7c server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:46.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.650+0000 7fefb7fff700 1 -- 172.21.15.49:0/2786814333 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefbc004160 con 0x7fefc0105f00 2023-07-20T17:42:46.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.651+0000 7fefb7fff700 1 -- 172.21.15.49:0/2786814333 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fefbc0042c0 con 0x7fefc0105f00 2023-07-20T17:42:46.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.651+0000 7fefb7fff700 1 -- 172.21.15.49:0/2786814333 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefbc0119b0 con 0x7fefc0105f00 2023-07-20T17:42:46.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.651+0000 7fefc78e1700 1 -- 172.21.15.49:0/2786814333 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 msgr2=0x7fefc0106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:46.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.651+0000 7fefc78e1700 1 --2- 172.21.15.49:0/2786814333 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0106340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0098a0 tx=0x7fefbc009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:46.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.651+0000 7fefc78e1700 1 -- 172.21.15.49:0/2786814333 shutdown_connections 2023-07-20T17:42:46.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.651+0000 7fefc78e1700 1 --2- 172.21.15.49:0/2786814333 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0106340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:46.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.651+0000 7fefc78e1700 1 -- 172.21.15.49:0/2786814333 >> 172.21.15.49:0/2786814333 conn(0x7fefc0101550 msgr2=0x7fefc0103970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:46.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.651+0000 7fefc78e1700 1 -- 172.21.15.49:0/2786814333 shutdown_connections 2023-07-20T17:42:46.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.651+0000 7fefc78e1700 1 -- 172.21.15.49:0/2786814333 wait complete. 2023-07-20T17:42:46.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.652+0000 7fefc78e1700 1 Processor -- start 2023-07-20T17:42:46.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.652+0000 7fefc78e1700 1 -- start start 2023-07-20T17:42:46.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.652+0000 7fefc78e1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:46.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.652+0000 7fefc78e1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc019a110 con 0x7fefc0105f00 2023-07-20T17:42:46.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.653+0000 7fefc567d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:46.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.653+0000 7fefc567d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35378/0 (socket says 172.21.15.49:35378) 2023-07-20T17:42:46.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.653+0000 7fefc567d700 1 -- 172.21.15.49:0/1396125911 learned_addr learned my addr 172.21.15.49:0/1396125911 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:46.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.653+0000 7fefc567d700 1 -- 172.21.15.49:0/1396125911 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc009550 con 0x7fefc0105f00 2023-07-20T17:42:46.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.653+0000 7fefc567d700 1 --2- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0199bd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fefbc003e60 tx=0x7fefbc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:46.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.653+0000 7fefb67fc700 1 -- 172.21.15.49:0/1396125911 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefbc003970 con 0x7fefc0105f00 2023-07-20T17:42:46.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.653+0000 7fefb67fc700 1 -- 172.21.15.49:0/1396125911 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fefbc003ad0 con 0x7fefc0105f00 2023-07-20T17:42:46.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.653+0000 7fefc78e1700 1 -- 172.21.15.49:0/1396125911 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc019a370 con 0x7fefc0105f00 2023-07-20T17:42:46.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.653+0000 7fefb67fc700 1 -- 172.21.15.49:0/1396125911 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefbc019590 con 0x7fefc0105f00 2023-07-20T17:42:46.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.653+0000 7fefc78e1700 1 -- 172.21.15.49:0/1396125911 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc019a7f0 con 0x7fefc0105f00 2023-07-20T17:42:46.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.654+0000 7fefb67fc700 1 -- 172.21.15.49:0/1396125911 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fefbc019800 con 0x7fefc0105f00 2023-07-20T17:42:46.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.654+0000 7fefb67fc700 1 --2- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fefac038740 0x7fefac03ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:46.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.654+0000 7fefb67fc700 1 -- 172.21.15.49:0/1396125911 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fefbc04c810 con 0x7fefc0105f00 2023-07-20T17:42:46.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.654+0000 7fefc4e7c700 1 -- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fefac038740 msgr2=0x7fefac03ac00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/752420140 2023-07-20T17:42:46.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.654+0000 7fefc4e7c700 1 --2- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fefac038740 0x7fefac03ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T17:42:46.521 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.654+0000 7fefc78e1700 1 -- 172.21.15.49:0/1396125911 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc0107df0 con 0x7fefc0105f00 2023-07-20T17:42:46.521 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.659+0000 7fefb67fc700 1 -- 172.21.15.49:0/1396125911 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fefbc019b50 con 0x7fefc0105f00 2023-07-20T17:42:46.521 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.855+0000 7fefc4e7c700 1 -- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fefac038740 msgr2=0x7fefac03ac00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/752420140 2023-07-20T17:42:46.521 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:45.855+0000 7fefc4e7c700 1 --2- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fefac038740 0x7fefac03ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T17:42:46.521 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.121+0000 7fefc78e1700 1 -- 172.21.15.49:0/1396125911 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fefc019d770 con 0x7fefc0105f00 2023-07-20T17:42:46.521 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.121+0000 7fefb67fc700 1 -- 172.21.15.49:0/1396125911 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fefbc018b40 con 0x7fefc0105f00 2023-07-20T17:42:46.521 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.123+0000 7fefc78e1700 1 -- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fefac038740 msgr2=0x7fefac03ac00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T17:42:46.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.123+0000 7fefc78e1700 1 --2- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fefac038740 0x7fefac03ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:46.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.123+0000 7fefc78e1700 1 -- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 msgr2=0x7fefc0199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:46.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.123+0000 7fefc78e1700 1 --2- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0199bd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fefbc003e60 tx=0x7fefbc011aa0 comp rx=0 tx=0).stop 2023-07-20T17:42:46.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.123+0000 7fefc78e1700 1 -- 172.21.15.49:0/1396125911 shutdown_connections 2023-07-20T17:42:46.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.123+0000 7fefc78e1700 1 --2- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7fefac038740 0x7fefac03ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:46.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.123+0000 7fefc78e1700 1 --2- 172.21.15.49:0/1396125911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefc0105f00 0x7fefc0199bd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:46.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.123+0000 7fefc78e1700 1 -- 172.21.15.49:0/1396125911 >> 172.21.15.49:0/1396125911 conn(0x7fefc0101550 msgr2=0x7fefc0102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:46.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.123+0000 7fefc78e1700 1 -- 172.21.15.49:0/1396125911 shutdown_connections 2023-07-20T17:42:46.523 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:46.123+0000 7fefc78e1700 1 -- 172.21.15.49:0/1396125911 wait complete. 2023-07-20T17:42:46.523 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for the mgr to restart... 2023-07-20T17:42:46.523 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr epoch 9... 2023-07-20T17:42:46.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:46 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1396125911' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T17:42:46.758 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:46 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:46.741+0000 7f044c205200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T17:42:47.738 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:47 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:47.431+0000 7f044c205200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T17:42:47.738 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:47 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:47.623+0000 7f044c205200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T17:42:48.915 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:48 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:48.584+0000 7f044c205200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T17:42:48.915 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:48 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:48.694+0000 7f044c205200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T17:42:49.544 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:48 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:48.914+0000 7f044c205200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T17:42:49.544 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:49.267+0000 7f044c205200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T17:42:49.988 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:49.544+0000 7f044c205200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T17:42:49.988 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:49.725+0000 7f044c205200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T17:42:50.488 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:50 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:50.053+0000 7f044c205200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T17:42:51.238 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:50 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:50.876+0000 7f044c205200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T17:42:51.238 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:51 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:51.034+0000 7f044c205200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T17:42:51.694 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:51 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:51.281+0000 7f044c205200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T17:42:51.694 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:51 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:51.418+0000 7f044c205200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T17:42:52.988 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:51 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:51.694+0000 7f044c205200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T17:42:52.988 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:42:52 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:42:52.731+0000 7f044c205200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T17:42:53.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: Active manager daemon x restarted 2023-07-20T17:42:53.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: Activating manager daemon x 2023-07-20T17:42:53.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: osdmap e3: 0 total, 0 up, 0 in 2023-07-20T17:42:53.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: mgrmap e10: x(active, starting, since 0.00360476s) 2023-07-20T17:42:53.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:42:53.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T17:42:53.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T17:42:53.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T17:42:53.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T17:42:53.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: Manager daemon x is now available 2023-07-20T17:42:53.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:42:53.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:42:53.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T17:42:53.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T17:42:54.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2023-07-20T17:42:54.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-20T17:42:54.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T17:42:54.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2023-07-20T17:42:54.174 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.049+0000 7f319c6e8700 1 Processor -- start 2023-07-20T17:42:54.175 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.049+0000 7f319c6e8700 1 -- start start 2023-07-20T17:42:54.175 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.049+0000 7f319c6e8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194104670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:54.175 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.049+0000 7f319c6e8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3194104c40 con 0x7f3194104250 2023-07-20T17:42:54.175 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.050+0000 7f319a484700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:54.175 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.050+0000 7f319a484700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194104670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35386/0 (socket says 172.21.15.49:35386) 2023-07-20T17:42:54.175 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.050+0000 7f319a484700 1 -- 172.21.15.49:0/4266963881 learned_addr learned my addr 172.21.15.49:0/4266963881 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:54.176 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.050+0000 7f319a484700 1 -- 172.21.15.49:0/4266963881 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31941071e0 con 0x7f3194104250 2023-07-20T17:42:54.176 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.050+0000 7f319a484700 1 --2- 172.21.15.49:0/4266963881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194104670 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f31840098a0 tx=0x7f3184009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868728658cbff360 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:54.176 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.050+0000 7f3199482700 1 -- 172.21.15.49:0/4266963881 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3184004160 con 0x7f3194104250 2023-07-20T17:42:54.176 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.050+0000 7f3199482700 1 -- 172.21.15.49:0/4266963881 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31840042c0 con 0x7f3194104250 2023-07-20T17:42:54.176 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.050+0000 7f3199482700 1 -- 172.21.15.49:0/4266963881 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31840119b0 con 0x7f3194104250 2023-07-20T17:42:54.176 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.051+0000 7f319c6e8700 1 -- 172.21.15.49:0/4266963881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 msgr2=0x7f3194104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:54.177 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.051+0000 7f319c6e8700 1 --2- 172.21.15.49:0/4266963881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194104670 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f31840098a0 tx=0x7f3184009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:54.177 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.051+0000 7f319c6e8700 1 -- 172.21.15.49:0/4266963881 shutdown_connections 2023-07-20T17:42:54.177 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.051+0000 7f319c6e8700 1 --2- 172.21.15.49:0/4266963881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194104670 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:54.177 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.051+0000 7f319c6e8700 1 -- 172.21.15.49:0/4266963881 >> 172.21.15.49:0/4266963881 conn(0x7f31940ffbc0 msgr2=0x7f3194101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:54.177 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.051+0000 7f319c6e8700 1 -- 172.21.15.49:0/4266963881 shutdown_connections 2023-07-20T17:42:54.177 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.051+0000 7f319c6e8700 1 -- 172.21.15.49:0/4266963881 wait complete. 2023-07-20T17:42:54.178 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.052+0000 7f319c6e8700 1 Processor -- start 2023-07-20T17:42:54.178 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.052+0000 7f319c6e8700 1 -- start start 2023-07-20T17:42:54.178 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f319c6e8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194106b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:54.178 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f319c6e8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3194107080 con 0x7f3194104250 2023-07-20T17:42:54.178 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f319a484700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194106b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:54.178 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f319a484700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194106b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35388/0 (socket says 172.21.15.49:35388) 2023-07-20T17:42:54.178 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f319a484700 1 -- 172.21.15.49:0/2485579971 learned_addr learned my addr 172.21.15.49:0/2485579971 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:54.179 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f319a484700 1 -- 172.21.15.49:0/2485579971 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3184009550 con 0x7f3194104250 2023-07-20T17:42:54.179 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f319a484700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194106b40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3184003e60 tx=0x7f3184011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:54.179 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3184003a60 con 0x7f3194104250 2023-07-20T17:42:54.179 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3184003bc0 con 0x7f3194104250 2023-07-20T17:42:54.179 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31840196c0 con 0x7f3194104250 2023-07-20T17:42:54.179 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f319c6e8700 1 -- 172.21.15.49:0/2485579971 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3194105190 con 0x7f3194104250 2023-07-20T17:42:54.179 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.053+0000 7f319c6e8700 1 -- 172.21.15.49:0/2485579971 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31941055b0 con 0x7f3194104250 2023-07-20T17:42:54.179 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.054+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f3184019930 con 0x7f3194104250 2023-07-20T17:42:54.180 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.055+0000 7f318b7fe700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 0x7f318003ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:54.180 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.055+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 --> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f318003b360 con 0x7f3180038790 2023-07-20T17:42:54.180 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.055+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f318404cb70 con 0x7f3194104250 2023-07-20T17:42:54.180 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.055+0000 7f3199c83700 1 -- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 msgr2=0x7f318003ac50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/752420140 2023-07-20T17:42:54.180 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.055+0000 7f3199c83700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 0x7f318003ac50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T17:42:54.180 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.256+0000 7f3199c83700 1 -- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 msgr2=0x7f318003ac50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/752420140 2023-07-20T17:42:54.180 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.256+0000 7f3199c83700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 0x7f318003ac50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T17:42:54.181 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.657+0000 7f3199c83700 1 -- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 msgr2=0x7f318003ac50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/752420140 2023-07-20T17:42:54.181 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:47.657+0000 7f3199c83700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 0x7f318003ac50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T17:42:54.181 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:48.458+0000 7f3199c83700 1 -- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 msgr2=0x7f318003ac50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/752420140 2023-07-20T17:42:54.181 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:48.458+0000 7f3199c83700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 0x7f318003ac50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T17:42:54.181 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:50.059+0000 7f3199c83700 1 -- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 msgr2=0x7f318003ac50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/752420140 2023-07-20T17:42:54.181 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:50.059+0000 7f3199c83700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 0x7f318003ac50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T17:42:54.181 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:52.747+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f318400fd80 con 0x7f3194104250 2023-07-20T17:42:54.181 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:52.747+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 msgr2=0x7f318003ac50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T17:42:54.182 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:52.747+0000 7f318b7fe700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 0x7f318003ac50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:54.182 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.749+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f318404e5b0 con 0x7f3194104250 2023-07-20T17:42:54.182 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.749+0000 7f318b7fe700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f318003bfe0 0x7f318003e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:54.182 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.749+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f318003b360 con 0x7f318003bfe0 2023-07-20T17:42:54.182 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.749+0000 7f3199c83700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f318003bfe0 0x7f318003e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:54.182 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.750+0000 7f3199c83700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f318003bfe0 0x7f318003e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3190003a10 tx=0x7f3190009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:54.182 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.750+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f318003b360 con 0x7f318003bfe0 2023-07-20T17:42:54.182 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.761+0000 7f319c6e8700 1 -- 172.21.15.49:0/2485579971 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f31941066e0 con 0x7f318003bfe0 2023-07-20T17:42:54.183 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.761+0000 7f318b7fe700 1 -- 172.21.15.49:0/2485579971 <== mgr.14152 v2:172.21.15.49:6800/971762269 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f31941066e0 con 0x7f318003bfe0 2023-07-20T17:42:54.183 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.761+0000 7f319c6e8700 1 -- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f318003bfe0 msgr2=0x7f318003e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:54.183 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.761+0000 7f319c6e8700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f318003bfe0 0x7f318003e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3190003a10 tx=0x7f3190009270 comp rx=0 tx=0).stop 2023-07-20T17:42:54.183 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.761+0000 7f319c6e8700 1 -- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 msgr2=0x7f3194106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:54.183 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.761+0000 7f319c6e8700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194106b40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3184003e60 tx=0x7f3184011a30 comp rx=0 tx=0).stop 2023-07-20T17:42:54.183 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.761+0000 7f319c6e8700 1 -- 172.21.15.49:0/2485579971 shutdown_connections 2023-07-20T17:42:54.183 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.761+0000 7f319c6e8700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f318003bfe0 0x7f318003e3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:54.183 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.761+0000 7f319c6e8700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3194104250 0x7f3194106b40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:54.184 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.761+0000 7f319c6e8700 1 --2- 172.21.15.49:0/2485579971 >> [v2:172.21.15.49:6800/752420140,v1:172.21.15.49:6801/752420140] conn(0x7f3180038790 0x7f318003ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:54.184 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.761+0000 7f319c6e8700 1 -- 172.21.15.49:0/2485579971 >> 172.21.15.49:0/2485579971 conn(0x7f31940ffbc0 msgr2=0x7f31941009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:54.184 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.762+0000 7f319c6e8700 1 -- 172.21.15.49:0/2485579971 shutdown_connections 2023-07-20T17:42:54.184 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:53.762+0000 7f319c6e8700 1 -- 172.21.15.49:0/2485579971 wait complete. 2023-07-20T17:42:54.184 INFO:teuthology.orchestra.run.smithi049.stdout:mgr epoch 9 is available 2023-07-20T17:42:54.184 INFO:teuthology.orchestra.run.smithi049.stdout:Generating a dashboard self-signed certificate... 2023-07-20T17:42:54.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:54 smithi049 ceph-mon[96530]: [20/Jul/2023:17:42:53] ENGINE Bus STARTING 2023-07-20T17:42:54.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:54 smithi049 ceph-mon[96530]: [20/Jul/2023:17:42:53] ENGINE Serving on https://172.21.15.49:7150 2023-07-20T17:42:54.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:54 smithi049 ceph-mon[96530]: [20/Jul/2023:17:42:53] ENGINE Serving on http://172.21.15.49:8765 2023-07-20T17:42:54.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:54 smithi049 ceph-mon[96530]: [20/Jul/2023:17:42:53] ENGINE Bus STARTED 2023-07-20T17:42:54.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:54 smithi049 ceph-mon[96530]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T17:42:54.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:54 smithi049 ceph-mon[96530]: mgrmap e11: x(active, since 1.00636s) 2023-07-20T17:42:55.770 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-20T17:42:55.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.892+0000 7f222e410700 1 Processor -- start 2023-07-20T17:42:55.771 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.892+0000 7f222e410700 1 -- start start 2023-07-20T17:42:55.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.892+0000 7f222e410700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228108dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:55.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.892+0000 7f222e410700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2228079eb0 con 0x7f22281069d0 2023-07-20T17:42:55.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.893+0000 7f2227fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228108dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:55.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.893+0000 7f2227fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228108dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35442/0 (socket says 172.21.15.49:35442) 2023-07-20T17:42:55.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.893+0000 7f2227fff700 1 -- 172.21.15.49:0/3103277532 learned_addr learned my addr 172.21.15.49:0/3103277532 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:55.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.893+0000 7f2227fff700 1 -- 172.21.15.49:0/3103277532 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2228109300 con 0x7f22281069d0 2023-07-20T17:42:55.772 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.893+0000 7f2227fff700 1 --2- 172.21.15.49:0/3103277532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228108dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f22180098a0 tx=0x7f2218009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64f58d84d6a46df server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:55.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.893+0000 7f2226ffd700 1 -- 172.21.15.49:0/3103277532 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2218004160 con 0x7f22281069d0 2023-07-20T17:42:55.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.894+0000 7f2226ffd700 1 -- 172.21.15.49:0/3103277532 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22180042c0 con 0x7f22281069d0 2023-07-20T17:42:55.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.894+0000 7f2226ffd700 1 -- 172.21.15.49:0/3103277532 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22180119b0 con 0x7f22281069d0 2023-07-20T17:42:55.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.894+0000 7f222e410700 1 -- 172.21.15.49:0/3103277532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 msgr2=0x7f2228108dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:55.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.894+0000 7f222e410700 1 --2- 172.21.15.49:0/3103277532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228108dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f22180098a0 tx=0x7f2218009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:55.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.894+0000 7f222e410700 1 -- 172.21.15.49:0/3103277532 shutdown_connections 2023-07-20T17:42:55.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.894+0000 7f222e410700 1 --2- 172.21.15.49:0/3103277532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228108dc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:55.773 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.894+0000 7f222e410700 1 -- 172.21.15.49:0/3103277532 >> 172.21.15.49:0/3103277532 conn(0x7f2228100400 msgr2=0x7f2228102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:55.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.894+0000 7f222e410700 1 -- 172.21.15.49:0/3103277532 shutdown_connections 2023-07-20T17:42:55.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.894+0000 7f222e410700 1 -- 172.21.15.49:0/3103277532 wait complete. 2023-07-20T17:42:55.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.895+0000 7f222e410700 1 Processor -- start 2023-07-20T17:42:55.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.895+0000 7f222e410700 1 -- start start 2023-07-20T17:42:55.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f222e410700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:55.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f222e410700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222819a190 con 0x7f22281069d0 2023-07-20T17:42:55.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f2227fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:55.774 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f2227fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35444/0 (socket says 172.21.15.49:35444) 2023-07-20T17:42:55.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f2227fff700 1 -- 172.21.15.49:0/2348032228 learned_addr learned my addr 172.21.15.49:0/2348032228 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:55.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f2227fff700 1 -- 172.21.15.49:0/2348032228 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2218009550 con 0x7f22281069d0 2023-07-20T17:42:55.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f2227fff700 1 --2- 172.21.15.49:0/2348032228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228199c50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2218003f00 tx=0x7f2218011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:55.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f22257fa700 1 -- 172.21.15.49:0/2348032228 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2218003b20 con 0x7f22281069d0 2023-07-20T17:42:55.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f22257fa700 1 -- 172.21.15.49:0/2348032228 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2218003c80 con 0x7f22281069d0 2023-07-20T17:42:55.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f222e410700 1 -- 172.21.15.49:0/2348032228 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222819a390 con 0x7f22281069d0 2023-07-20T17:42:55.775 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f22257fa700 1 -- 172.21.15.49:0/2348032228 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2218019770 con 0x7f22281069d0 2023-07-20T17:42:55.776 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.896+0000 7f222e410700 1 -- 172.21.15.49:0/2348032228 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222819a7b0 con 0x7f22281069d0 2023-07-20T17:42:55.776 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.897+0000 7f22257fa700 1 -- 172.21.15.49:0/2348032228 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f2218019a30 con 0x7f22281069d0 2023-07-20T17:42:55.776 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.898+0000 7f22257fa700 1 --2- 172.21.15.49:0/2348032228 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f2210038630 0x7f221003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:55.776 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.898+0000 7f22257fa700 1 -- 172.21.15.49:0/2348032228 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f221804c8f0 con 0x7f22281069d0 2023-07-20T17:42:55.777 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.898+0000 7f22277fe700 1 --2- 172.21.15.49:0/2348032228 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f2210038630 0x7f221003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:55.777 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.898+0000 7f222e410700 1 -- 172.21.15.49:0/2348032228 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2228061da0 con 0x7f22281069d0 2023-07-20T17:42:55.777 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.898+0000 7f22277fe700 1 --2- 172.21.15.49:0/2348032228 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f2210038630 0x7f221003aaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f221c006fd0 tx=0x7f221c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:55.777 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:54.903+0000 7f22257fa700 1 -- 172.21.15.49:0/2348032228 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2218018800 con 0x7f22281069d0 2023-07-20T17:42:55.777 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.283+0000 7f222e410700 1 -- 172.21.15.49:0/2348032228 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f22281026e0 con 0x7f2210038630 2023-07-20T17:42:55.777 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.329+0000 7f22257fa700 1 -- 172.21.15.49:0/2348032228 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f22281026e0 con 0x7f2210038630 2023-07-20T17:42:55.777 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.331+0000 7f222e410700 1 -- 172.21.15.49:0/2348032228 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f2210038630 msgr2=0x7f221003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:55.778 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.331+0000 7f222e410700 1 --2- 172.21.15.49:0/2348032228 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f2210038630 0x7f221003aaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f221c006fd0 tx=0x7f221c006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:55.778 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.331+0000 7f222e410700 1 -- 172.21.15.49:0/2348032228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 msgr2=0x7f2228199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:55.778 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.331+0000 7f222e410700 1 --2- 172.21.15.49:0/2348032228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228199c50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2218003f00 tx=0x7f2218011a30 comp rx=0 tx=0).stop 2023-07-20T17:42:55.778 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.331+0000 7f222e410700 1 -- 172.21.15.49:0/2348032228 shutdown_connections 2023-07-20T17:42:55.778 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.331+0000 7f222e410700 1 --2- 172.21.15.49:0/2348032228 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f2210038630 0x7f221003aaf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:55.778 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.331+0000 7f222e410700 1 --2- 172.21.15.49:0/2348032228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22281069d0 0x7f2228199c50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:55.778 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.331+0000 7f222e410700 1 -- 172.21.15.49:0/2348032228 >> 172.21.15.49:0/2348032228 conn(0x7f2228100400 msgr2=0x7f2228101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:55.779 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.331+0000 7f222e410700 1 -- 172.21.15.49:0/2348032228 shutdown_connections 2023-07-20T17:42:55.779 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:55.331+0000 7f222e410700 1 -- 172.21.15.49:0/2348032228 wait complete. 2023-07-20T17:42:55.779 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial admin user... 2023-07-20T17:42:55.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:55 smithi049 ceph-mon[96530]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T17:42:55.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:55 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:42:55.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:55 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:42:55.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:55 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:42:55.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:55 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:42:56.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:56 smithi049 ceph-mon[96530]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:42:56.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:56 smithi049 ceph-mon[96530]: mgrmap e12: x(active, since 3s) 2023-07-20T17:42:57.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$J9P150BGID5nW.HXZAbzXOX3UtKZk1nrED91XwYXjYL220hmJa31K", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689874976, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-20T17:42:57.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.269+0000 7fbf4b3d7700 1 Processor -- start 2023-07-20T17:42:57.519 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.269+0000 7fbf4b3d7700 1 -- start start 2023-07-20T17:42:57.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.269+0000 7fbf4b3d7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44104670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:57.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.269+0000 7fbf4b3d7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf44104c40 con 0x7fbf44104250 2023-07-20T17:42:57.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.269+0000 7fbf49173700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:57.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.269+0000 7fbf49173700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44104670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35448/0 (socket says 172.21.15.49:35448) 2023-07-20T17:42:57.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.269+0000 7fbf49173700 1 -- 172.21.15.49:0/1555076098 learned_addr learned my addr 172.21.15.49:0/1555076098 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:57.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.270+0000 7fbf49173700 1 -- 172.21.15.49:0/1555076098 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf441071e0 con 0x7fbf44104250 2023-07-20T17:42:57.520 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.270+0000 7fbf49173700 1 --2- 172.21.15.49:0/1555076098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44104670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbf400098a0 tx=0x7fbf40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f49fa0f66599f52 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:57.521 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.270+0000 7fbf3bfff700 1 -- 172.21.15.49:0/1555076098 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf40004160 con 0x7fbf44104250 2023-07-20T17:42:57.521 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.270+0000 7fbf3bfff700 1 -- 172.21.15.49:0/1555076098 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf400042c0 con 0x7fbf44104250 2023-07-20T17:42:57.521 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.270+0000 7fbf3bfff700 1 -- 172.21.15.49:0/1555076098 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf400119b0 con 0x7fbf44104250 2023-07-20T17:42:57.521 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.271+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1555076098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 msgr2=0x7fbf44104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:57.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.271+0000 7fbf4b3d7700 1 --2- 172.21.15.49:0/1555076098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44104670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbf400098a0 tx=0x7fbf40009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:57.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.271+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1555076098 shutdown_connections 2023-07-20T17:42:57.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.271+0000 7fbf4b3d7700 1 --2- 172.21.15.49:0/1555076098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44104670 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:57.522 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.271+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1555076098 >> 172.21.15.49:0/1555076098 conn(0x7fbf440ffbc0 msgr2=0x7fbf44101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:57.523 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.271+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1555076098 shutdown_connections 2023-07-20T17:42:57.523 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.271+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1555076098 wait complete. 2023-07-20T17:42:57.523 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.271+0000 7fbf4b3d7700 1 Processor -- start 2023-07-20T17:42:57.523 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.271+0000 7fbf4b3d7700 1 -- start start 2023-07-20T17:42:57.523 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf4b3d7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44106b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:57.524 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf4b3d7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf44107080 con 0x7fbf44104250 2023-07-20T17:42:57.524 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf49173700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44106b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:57.524 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf49173700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44106b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35450/0 (socket says 172.21.15.49:35450) 2023-07-20T17:42:57.524 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf49173700 1 -- 172.21.15.49:0/1515365759 learned_addr learned my addr 172.21.15.49:0/1515365759 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:57.525 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf49173700 1 -- 172.21.15.49:0/1515365759 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf40009550 con 0x7fbf44104250 2023-07-20T17:42:57.525 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf49173700 1 --2- 172.21.15.49:0/1515365759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44106b40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbf40006af0 tx=0x7fbf40011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:57.525 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf3a7fc700 1 -- 172.21.15.49:0/1515365759 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf40003a40 con 0x7fbf44104250 2023-07-20T17:42:57.525 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1515365759 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf44105190 con 0x7fbf44104250 2023-07-20T17:42:57.526 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf3a7fc700 1 -- 172.21.15.49:0/1515365759 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf40003ba0 con 0x7fbf44104250 2023-07-20T17:42:57.526 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf3a7fc700 1 -- 172.21.15.49:0/1515365759 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf40019690 con 0x7fbf44104250 2023-07-20T17:42:57.526 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.272+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1515365759 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf441055b0 con 0x7fbf44104250 2023-07-20T17:42:57.526 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.273+0000 7fbf3a7fc700 1 -- 172.21.15.49:0/1515365759 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fbf4001aa60 con 0x7fbf44104250 2023-07-20T17:42:57.527 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.274+0000 7fbf3a7fc700 1 --2- 172.21.15.49:0/1515365759 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbf30038710 0x7fbf3003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:57.527 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.274+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1515365759 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4410a410 con 0x7fbf44104250 2023-07-20T17:42:57.527 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.274+0000 7fbf3a7fc700 1 -- 172.21.15.49:0/1515365759 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbf4004cc30 con 0x7fbf44104250 2023-07-20T17:42:57.527 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.274+0000 7fbf48972700 1 --2- 172.21.15.49:0/1515365759 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbf30038710 0x7fbf3003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:57.528 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.274+0000 7fbf48972700 1 --2- 172.21.15.49:0/1515365759 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbf30038710 0x7fbf3003abd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbf34006fd0 tx=0x7fbf34006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:57.528 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.279+0000 7fbf3a7fc700 1 -- 172.21.15.49:0/1515365759 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbf40023980 con 0x7fbf44104250 2023-07-20T17:42:57.528 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.675+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1515365759 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fbf4410a600 con 0x7fbf30038710 2023-07-20T17:42:57.528 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.922+0000 7fbf3a7fc700 1 -- 172.21.15.49:0/1515365759 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fbf4410a600 con 0x7fbf30038710 2023-07-20T17:42:57.529 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.924+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1515365759 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbf30038710 msgr2=0x7fbf3003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:57.529 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.924+0000 7fbf4b3d7700 1 --2- 172.21.15.49:0/1515365759 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbf30038710 0x7fbf3003abd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbf34006fd0 tx=0x7fbf34006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:57.529 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.924+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1515365759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 msgr2=0x7fbf44106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:57.529 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.924+0000 7fbf4b3d7700 1 --2- 172.21.15.49:0/1515365759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44106b40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbf40006af0 tx=0x7fbf40011a10 comp rx=0 tx=0).stop 2023-07-20T17:42:57.529 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.924+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1515365759 shutdown_connections 2023-07-20T17:42:57.530 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.924+0000 7fbf4b3d7700 1 --2- 172.21.15.49:0/1515365759 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbf30038710 0x7fbf3003abd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:57.530 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.924+0000 7fbf4b3d7700 1 --2- 172.21.15.49:0/1515365759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf44104250 0x7fbf44106b40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:57.530 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.924+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1515365759 >> 172.21.15.49:0/1515365759 conn(0x7fbf440ffbc0 msgr2=0x7fbf441009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:57.531 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.924+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1515365759 shutdown_connections 2023-07-20T17:42:57.531 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:56.924+0000 7fbf4b3d7700 1 -- 172.21.15.49:0/1515365759 wait complete. 2023-07-20T17:42:57.531 INFO:teuthology.orchestra.run.smithi049.stdout:Fetching dashboard port number... 2023-07-20T17:42:57.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:57 smithi049 ceph-mon[96530]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:42:57.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:57 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:42:58.871 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 8443 2023-07-20T17:42:58.871 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.036+0000 7f513d549700 1 Processor -- start 2023-07-20T17:42:58.871 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.036+0000 7f513d549700 1 -- start start 2023-07-20T17:42:58.871 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.036+0000 7f513d549700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f5138104c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:58.872 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.036+0000 7f513d549700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5138079e50 con 0x7f5138104820 2023-07-20T17:42:58.872 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.037+0000 7f5136ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f5138104c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:58.872 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.037+0000 7f5136ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f5138104c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35454/0 (socket says 172.21.15.49:35454) 2023-07-20T17:42:58.872 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.037+0000 7f5136ffd700 1 -- 172.21.15.49:0/3647474569 learned_addr learned my addr 172.21.15.49:0/3647474569 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:58.872 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.037+0000 7f5136ffd700 1 -- 172.21.15.49:0/3647474569 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5138105180 con 0x7f5138104820 2023-07-20T17:42:58.872 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.037+0000 7f5136ffd700 1 --2- 172.21.15.49:0/3647474569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f5138104c40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f51280098a0 tx=0x7f5128009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7022b39babf238b7 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:58.872 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.037+0000 7f5135ffb700 1 -- 172.21.15.49:0/3647474569 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5128004160 con 0x7f5138104820 2023-07-20T17:42:58.873 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.037+0000 7f5135ffb700 1 -- 172.21.15.49:0/3647474569 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51280042c0 con 0x7f5138104820 2023-07-20T17:42:58.873 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.037+0000 7f5135ffb700 1 -- 172.21.15.49:0/3647474569 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51280119b0 con 0x7f5138104820 2023-07-20T17:42:58.873 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.038+0000 7f513d549700 1 -- 172.21.15.49:0/3647474569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 msgr2=0x7f5138104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:58.873 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.038+0000 7f513d549700 1 --2- 172.21.15.49:0/3647474569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f5138104c40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f51280098a0 tx=0x7f5128009bb0 comp rx=0 tx=0).stop 2023-07-20T17:42:58.873 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.038+0000 7f513d549700 1 -- 172.21.15.49:0/3647474569 shutdown_connections 2023-07-20T17:42:58.873 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.038+0000 7f513d549700 1 --2- 172.21.15.49:0/3647474569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f5138104c40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:58.873 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.038+0000 7f513d549700 1 -- 172.21.15.49:0/3647474569 >> 172.21.15.49:0/3647474569 conn(0x7f5138100400 msgr2=0x7f5138102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:58.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.038+0000 7f513d549700 1 -- 172.21.15.49:0/3647474569 shutdown_connections 2023-07-20T17:42:58.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.038+0000 7f513d549700 1 -- 172.21.15.49:0/3647474569 wait complete. 2023-07-20T17:42:58.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.039+0000 7f513d549700 1 Processor -- start 2023-07-20T17:42:58.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.039+0000 7f513d549700 1 -- start start 2023-07-20T17:42:58.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.039+0000 7f513d549700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f513819be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:58.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.039+0000 7f513d549700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513819c380 con 0x7f5138104820 2023-07-20T17:42:58.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.040+0000 7f5136ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f513819be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:58.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.040+0000 7f5136ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f513819be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35456/0 (socket says 172.21.15.49:35456) 2023-07-20T17:42:58.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.040+0000 7f5136ffd700 1 -- 172.21.15.49:0/1756104323 learned_addr learned my addr 172.21.15.49:0/1756104323 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:42:58.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.040+0000 7f5136ffd700 1 -- 172.21.15.49:0/1756104323 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5128009550 con 0x7f5138104820 2023-07-20T17:42:58.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.040+0000 7f5136ffd700 1 --2- 172.21.15.49:0/1756104323 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f513819be40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5128003f00 tx=0x7f5128011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:58.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.040+0000 7f511ffff700 1 -- 172.21.15.49:0/1756104323 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5128003b20 con 0x7f5138104820 2023-07-20T17:42:58.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.040+0000 7f513d549700 1 -- 172.21.15.49:0/1756104323 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513819c580 con 0x7f5138104820 2023-07-20T17:42:58.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.040+0000 7f511ffff700 1 -- 172.21.15.49:0/1756104323 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5128003c80 con 0x7f5138104820 2023-07-20T17:42:58.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.040+0000 7f511ffff700 1 -- 172.21.15.49:0/1756104323 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5128019770 con 0x7f5138104820 2023-07-20T17:42:58.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.040+0000 7f513d549700 1 -- 172.21.15.49:0/1756104323 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513819c9a0 con 0x7f5138104820 2023-07-20T17:42:58.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.041+0000 7f511ffff700 1 -- 172.21.15.49:0/1756104323 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f5128019a30 con 0x7f5138104820 2023-07-20T17:42:58.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.041+0000 7f511ffff700 1 --2- 172.21.15.49:0/1756104323 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5120038760 0x7f512003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:42:58.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.041+0000 7f511ffff700 1 -- 172.21.15.49:0/1756104323 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f512804cab0 con 0x7f5138104820 2023-07-20T17:42:58.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.042+0000 7f51367fc700 1 --2- 172.21.15.49:0/1756104323 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5120038760 0x7f512003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:42:58.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.042+0000 7f513d549700 1 -- 172.21.15.49:0/1756104323 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5138195710 con 0x7f5138104820 2023-07-20T17:42:58.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.042+0000 7f51367fc700 1 --2- 172.21.15.49:0/1756104323 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5120038760 0x7f512003ac20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f512c006fd0 tx=0x7f512c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:42:58.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.047+0000 7f511ffff700 1 -- 172.21.15.49:0/1756104323 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5128018800 con 0x7f5138104820 2023-07-20T17:42:58.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.407+0000 7f513d549700 1 -- 172.21.15.49:0/1756104323 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f513802cc20 con 0x7f5138104820 2023-07-20T17:42:58.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.408+0000 7f511ffff700 1 -- 172.21.15.49:0/1756104323 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f5128023e60 con 0x7f5138104820 2023-07-20T17:42:58.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.410+0000 7f513d549700 1 -- 172.21.15.49:0/1756104323 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5120038760 msgr2=0x7f512003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:58.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.410+0000 7f513d549700 1 --2- 172.21.15.49:0/1756104323 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5120038760 0x7f512003ac20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f512c006fd0 tx=0x7f512c006e40 comp rx=0 tx=0).stop 2023-07-20T17:42:58.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.410+0000 7f513d549700 1 -- 172.21.15.49:0/1756104323 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 msgr2=0x7f513819be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:42:58.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.410+0000 7f513d549700 1 --2- 172.21.15.49:0/1756104323 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f513819be40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5128003f00 tx=0x7f5128011a30 comp rx=0 tx=0).stop 2023-07-20T17:42:58.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.410+0000 7f513d549700 1 -- 172.21.15.49:0/1756104323 shutdown_connections 2023-07-20T17:42:58.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.410+0000 7f513d549700 1 --2- 172.21.15.49:0/1756104323 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5120038760 0x7f512003ac20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:58.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.410+0000 7f513d549700 1 --2- 172.21.15.49:0/1756104323 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5138104820 0x7f513819be40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:42:58.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.410+0000 7f513d549700 1 -- 172.21.15.49:0/1756104323 >> 172.21.15.49:0/1756104323 conn(0x7f5138100400 msgr2=0x7f51381903e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:42:58.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.410+0000 7f513d549700 1 -- 172.21.15.49:0/1756104323 shutdown_connections 2023-07-20T17:42:58.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:58.410+0000 7f513d549700 1 -- 172.21.15.49:0/1756104323 wait complete. 2023-07-20T17:42:58.879 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T17:42:58.879 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2023-07-20T17:42:58.884 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T17:42:58.884 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2023-07-20T17:42:58.885 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2023-07-20T17:42:58.885 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-20T17:42:58.886 INFO:teuthology.orchestra.run.smithi049.stdout:Ceph Dashboard is now available at: 2023-07-20T17:42:58.886 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:42:58.886 INFO:teuthology.orchestra.run.smithi049.stdout: URL: https://smithi049:8443/ 2023-07-20T17:42:58.887 INFO:teuthology.orchestra.run.smithi049.stdout: User: admin 2023-07-20T17:42:58.887 INFO:teuthology.orchestra.run.smithi049.stdout: Password: gpwnmw02wi 2023-07-20T17:42:58.887 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:42:58.887 INFO:teuthology.orchestra.run.smithi049.stdout:Saving cluster configuration to /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config directory 2023-07-20T17:42:58.887 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling autotune for osd_memory_target 2023-07-20T17:42:59.098 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:42:58 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1756104323' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-20T17:43:00.135 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.388+0000 7f75fbcfa700 1 Processor -- start 2023-07-20T17:43:00.135 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.388+0000 7f75fbcfa700 1 -- start start 2023-07-20T17:43:00.135 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.388+0000 7f75fbcfa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4108e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:00.136 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.388+0000 7f75fbcfa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f4079e70 con 0x7f75f4106a20 2023-07-20T17:43:00.136 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.389+0000 7f75f9a96700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4108e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:00.136 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.389+0000 7f75f9a96700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4108e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35464/0 (socket says 172.21.15.49:35464) 2023-07-20T17:43:00.136 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.389+0000 7f75f9a96700 1 -- 172.21.15.49:0/3146366964 learned_addr learned my addr 172.21.15.49:0/3146366964 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:00.137 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.389+0000 7f75f9a96700 1 -- 172.21.15.49:0/3146366964 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4109350 con 0x7f75f4106a20 2023-07-20T17:43:00.137 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.389+0000 7f75f9a96700 1 --2- 172.21.15.49:0/3146366964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4108e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f75f00098a0 tx=0x7f75f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1594de1d6aea0f8 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:00.137 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.389+0000 7f75f8a94700 1 -- 172.21.15.49:0/3146366964 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75f0004160 con 0x7f75f4106a20 2023-07-20T17:43:00.137 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.389+0000 7f75f8a94700 1 -- 172.21.15.49:0/3146366964 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75f00042c0 con 0x7f75f4106a20 2023-07-20T17:43:00.137 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.389+0000 7f75f8a94700 1 -- 172.21.15.49:0/3146366964 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75f00119b0 con 0x7f75f4106a20 2023-07-20T17:43:00.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.390+0000 7f75fbcfa700 1 -- 172.21.15.49:0/3146366964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 msgr2=0x7f75f4108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:00.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.390+0000 7f75fbcfa700 1 --2- 172.21.15.49:0/3146366964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4108e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f75f00098a0 tx=0x7f75f0009bb0 comp rx=0 tx=0).stop 2023-07-20T17:43:00.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.390+0000 7f75fbcfa700 1 -- 172.21.15.49:0/3146366964 shutdown_connections 2023-07-20T17:43:00.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.390+0000 7f75fbcfa700 1 --2- 172.21.15.49:0/3146366964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4108e10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:00.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.390+0000 7f75fbcfa700 1 -- 172.21.15.49:0/3146366964 >> 172.21.15.49:0/3146366964 conn(0x7f75f4100430 msgr2=0x7f75f4102850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:00.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.390+0000 7f75fbcfa700 1 -- 172.21.15.49:0/3146366964 shutdown_connections 2023-07-20T17:43:00.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.390+0000 7f75fbcfa700 1 -- 172.21.15.49:0/3146366964 wait complete. 2023-07-20T17:43:00.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.391+0000 7f75fbcfa700 1 Processor -- start 2023-07-20T17:43:00.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.391+0000 7f75fbcfa700 1 -- start start 2023-07-20T17:43:00.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.391+0000 7f75fbcfa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4106380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:00.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.391+0000 7f75fbcfa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f41068c0 con 0x7f75f4106a20 2023-07-20T17:43:00.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.391+0000 7f75f9a96700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4106380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:00.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.391+0000 7f75f9a96700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4106380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35466/0 (socket says 172.21.15.49:35466) 2023-07-20T17:43:00.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.391+0000 7f75f9a96700 1 -- 172.21.15.49:0/569877669 learned_addr learned my addr 172.21.15.49:0/569877669 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:00.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.391+0000 7f75f9a96700 1 -- 172.21.15.49:0/569877669 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f0009550 con 0x7f75f4106a20 2023-07-20T17:43:00.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.391+0000 7f75f9a96700 1 --2- 172.21.15.49:0/569877669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4106380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f75f0003f00 tx=0x7f75f0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:00.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.392+0000 7f75eaffd700 1 -- 172.21.15.49:0/569877669 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75f0003b20 con 0x7f75f4106a20 2023-07-20T17:43:00.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.392+0000 7f75eaffd700 1 -- 172.21.15.49:0/569877669 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75f0003c80 con 0x7f75f4106a20 2023-07-20T17:43:00.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.392+0000 7f75fbcfa700 1 -- 172.21.15.49:0/569877669 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f41049d0 con 0x7f75f4106a20 2023-07-20T17:43:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.392+0000 7f75eaffd700 1 -- 172.21.15.49:0/569877669 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75f0019770 con 0x7f75f4106a20 2023-07-20T17:43:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.392+0000 7f75fbcfa700 1 -- 172.21.15.49:0/569877669 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f4104df0 con 0x7f75f4106a20 2023-07-20T17:43:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.392+0000 7f75eaffd700 1 -- 172.21.15.49:0/569877669 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f75f00199e0 con 0x7f75f4106a20 2023-07-20T17:43:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.393+0000 7f75eaffd700 1 --2- 172.21.15.49:0/569877669 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f75e00387b0 0x7f75e003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.393+0000 7f75eaffd700 1 -- 172.21.15.49:0/569877669 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f75f004cb40 con 0x7f75f4106a20 2023-07-20T17:43:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.393+0000 7f75f9295700 1 --2- 172.21.15.49:0/569877669 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f75e00387b0 0x7f75e003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.393+0000 7f75fbcfa700 1 -- 172.21.15.49:0/569877669 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f410a130 con 0x7f75f4106a20 2023-07-20T17:43:00.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.393+0000 7f75f9295700 1 --2- 172.21.15.49:0/569877669 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f75e00387b0 0x7f75e003ac70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75e4006fd0 tx=0x7f75e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:00.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.398+0000 7f75eaffd700 1 -- 172.21.15.49:0/569877669 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f75f0018b80 con 0x7f75f4106a20 2023-07-20T17:43:00.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.755+0000 7f75eaffd700 1 -- 172.21.15.49:0/569877669 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f75f004cd50 con 0x7f75f4106a20 2023-07-20T17:43:00.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.758+0000 7f75fbcfa700 1 -- 172.21.15.49:0/569877669 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f75f410a2f0 con 0x7f75f4106a20 2023-07-20T17:43:00.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.758+0000 7f75eaffd700 1 -- 172.21.15.49:0/569877669 <== mon.0 v2:172.21.15.49:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f75f00189a0 con 0x7f75f4106a20 2023-07-20T17:43:00.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.761+0000 7f75fbcfa700 1 -- 172.21.15.49:0/569877669 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f75e00387b0 msgr2=0x7f75e003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:00.143 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.761+0000 7f75fbcfa700 1 --2- 172.21.15.49:0/569877669 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f75e00387b0 0x7f75e003ac70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75e4006fd0 tx=0x7f75e4006e40 comp rx=0 tx=0).stop 2023-07-20T17:43:00.143 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.761+0000 7f75fbcfa700 1 -- 172.21.15.49:0/569877669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 msgr2=0x7f75f4106380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:00.143 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.761+0000 7f75fbcfa700 1 --2- 172.21.15.49:0/569877669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4106380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f75f0003f00 tx=0x7f75f0011a30 comp rx=0 tx=0).stop 2023-07-20T17:43:00.143 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.761+0000 7f75fbcfa700 1 -- 172.21.15.49:0/569877669 shutdown_connections 2023-07-20T17:43:00.143 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.761+0000 7f75fbcfa700 1 --2- 172.21.15.49:0/569877669 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f75e00387b0 0x7f75e003ac70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:00.143 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.761+0000 7f75fbcfa700 1 --2- 172.21.15.49:0/569877669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4106a20 0x7f75f4106380 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:00.143 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.761+0000 7f75fbcfa700 1 -- 172.21.15.49:0/569877669 >> 172.21.15.49:0/569877669 conn(0x7f75f4100430 msgr2=0x7f75f4101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:00.143 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.761+0000 7f75fbcfa700 1 -- 172.21.15.49:0/569877669 shutdown_connections 2023-07-20T17:43:00.144 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:42:59.761+0000 7f75fbcfa700 1 -- 172.21.15.49:0/569877669 wait complete. 2023-07-20T17:43:01.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:00 smithi049 ceph-mon[96530]: mgrmap e13: x(active, since 7s) 2023-07-20T17:43:01.682 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.645+0000 7fee1ca36700 1 Processor -- start 2023-07-20T17:43:01.683 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.645+0000 7fee1ca36700 1 -- start start 2023-07-20T17:43:01.683 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.645+0000 7fee1ca36700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18105280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:01.683 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.645+0000 7fee1ca36700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee181057c0 con 0x7fee18104e60 2023-07-20T17:43:01.683 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.645+0000 7fee1659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18105280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:01.683 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.645+0000 7fee1659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18105280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35470/0 (socket says 172.21.15.49:35470) 2023-07-20T17:43:01.683 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.645+0000 7fee1659c700 1 -- 172.21.15.49:0/1453498940 learned_addr learned my addr 172.21.15.49:0/1453498940 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:01.684 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.646+0000 7fee1659c700 1 -- 172.21.15.49:0/1453498940 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee18105900 con 0x7fee18104e60 2023-07-20T17:43:01.684 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.646+0000 7fee1659c700 1 --2- 172.21.15.49:0/1453498940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18105280 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0098a0 tx=0x7fee0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97436fbd3baeabfb server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:01.684 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.646+0000 7fee1559a700 1 -- 172.21.15.49:0/1453498940 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee0c004160 con 0x7fee18104e60 2023-07-20T17:43:01.684 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.646+0000 7fee1559a700 1 -- 172.21.15.49:0/1453498940 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee0c0042c0 con 0x7fee18104e60 2023-07-20T17:43:01.684 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.646+0000 7fee1559a700 1 -- 172.21.15.49:0/1453498940 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee0c0119b0 con 0x7fee18104e60 2023-07-20T17:43:01.684 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.647+0000 7fee1ca36700 1 -- 172.21.15.49:0/1453498940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 msgr2=0x7fee18105280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:01.684 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.647+0000 7fee1ca36700 1 --2- 172.21.15.49:0/1453498940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18105280 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0098a0 tx=0x7fee0c009bb0 comp rx=0 tx=0).stop 2023-07-20T17:43:01.685 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.647+0000 7fee1ca36700 1 -- 172.21.15.49:0/1453498940 shutdown_connections 2023-07-20T17:43:01.685 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.647+0000 7fee1ca36700 1 --2- 172.21.15.49:0/1453498940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18105280 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:01.685 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.647+0000 7fee1ca36700 1 -- 172.21.15.49:0/1453498940 >> 172.21.15.49:0/1453498940 conn(0x7fee18100400 msgr2=0x7fee18102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:01.685 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.647+0000 7fee1ca36700 1 -- 172.21.15.49:0/1453498940 shutdown_connections 2023-07-20T17:43:01.685 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.647+0000 7fee1ca36700 1 -- 172.21.15.49:0/1453498940 wait complete. 2023-07-20T17:43:01.685 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.648+0000 7fee1ca36700 1 Processor -- start 2023-07-20T17:43:01.685 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.648+0000 7fee1ca36700 1 -- start start 2023-07-20T17:43:01.686 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.648+0000 7fee1ca36700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18079b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:01.686 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.648+0000 7fee1ca36700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1819c980 con 0x7fee18104e60 2023-07-20T17:43:01.686 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.648+0000 7fee1659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18079b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:01.686 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.648+0000 7fee1659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18079b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35472/0 (socket says 172.21.15.49:35472) 2023-07-20T17:43:01.686 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.648+0000 7fee1659c700 1 -- 172.21.15.49:0/2915485273 learned_addr learned my addr 172.21.15.49:0/2915485273 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:01.686 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.648+0000 7fee1659c700 1 -- 172.21.15.49:0/2915485273 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c009550 con 0x7fee18104e60 2023-07-20T17:43:01.686 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.648+0000 7fee1659c700 1 --2- 172.21.15.49:0/2915485273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18079b30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fee0c003f00 tx=0x7fee0c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:01.686 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.648+0000 7fee077fe700 1 -- 172.21.15.49:0/2915485273 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee0c003b20 con 0x7fee18104e60 2023-07-20T17:43:01.687 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.649+0000 7fee077fe700 1 -- 172.21.15.49:0/2915485273 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee0c003c80 con 0x7fee18104e60 2023-07-20T17:43:01.687 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.649+0000 7fee1ca36700 1 -- 172.21.15.49:0/2915485273 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1819cac0 con 0x7fee18104e60 2023-07-20T17:43:01.687 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.649+0000 7fee077fe700 1 -- 172.21.15.49:0/2915485273 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee0c019770 con 0x7fee18104e60 2023-07-20T17:43:01.687 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.649+0000 7fee1ca36700 1 -- 172.21.15.49:0/2915485273 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1819cee0 con 0x7fee18104e60 2023-07-20T17:43:01.687 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.649+0000 7fee077fe700 1 -- 172.21.15.49:0/2915485273 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fee0c019a30 con 0x7fee18104e60 2023-07-20T17:43:01.687 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.650+0000 7fee077fe700 1 --2- 172.21.15.49:0/2915485273 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee00038800 0x7fee0003acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.650+0000 7fee077fe700 1 -- 172.21.15.49:0/2915485273 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fee0c04ce90 con 0x7fee18104e60 2023-07-20T17:43:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.650+0000 7fee15d9b700 1 --2- 172.21.15.49:0/2915485273 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee00038800 0x7fee0003acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.650+0000 7fee1ca36700 1 -- 172.21.15.49:0/2915485273 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee18061da0 con 0x7fee18104e60 2023-07-20T17:43:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.650+0000 7fee15d9b700 1 --2- 172.21.15.49:0/2915485273 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee00038800 0x7fee0003acc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fee08006fd0 tx=0x7fee08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:00.655+0000 7fee077fe700 1 -- 172.21.15.49:0/2915485273 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fee0c018c10 con 0x7fee18104e60 2023-07-20T17:43:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.227+0000 7fee1ca36700 1 -- 172.21.15.49:0/2915485273 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fee18106cd0 con 0x7fee18104e60 2023-07-20T17:43:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.229+0000 7fee077fe700 1 -- 172.21.15.49:0/2915485273 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fee0c018800 con 0x7fee18104e60 2023-07-20T17:43:01.689 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.231+0000 7fee1ca36700 1 -- 172.21.15.49:0/2915485273 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee00038800 msgr2=0x7fee0003acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:01.689 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.231+0000 7fee1ca36700 1 --2- 172.21.15.49:0/2915485273 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee00038800 0x7fee0003acc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fee08006fd0 tx=0x7fee08006e40 comp rx=0 tx=0).stop 2023-07-20T17:43:01.689 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.231+0000 7fee1ca36700 1 -- 172.21.15.49:0/2915485273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 msgr2=0x7fee18079b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:01.689 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.231+0000 7fee1ca36700 1 --2- 172.21.15.49:0/2915485273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18079b30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fee0c003f00 tx=0x7fee0c011a30 comp rx=0 tx=0).stop 2023-07-20T17:43:01.689 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.231+0000 7fee1ca36700 1 -- 172.21.15.49:0/2915485273 shutdown_connections 2023-07-20T17:43:01.689 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.231+0000 7fee1ca36700 1 --2- 172.21.15.49:0/2915485273 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee00038800 0x7fee0003acc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:01.689 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.231+0000 7fee1ca36700 1 --2- 172.21.15.49:0/2915485273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee18104e60 0x7fee18079b30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:01.690 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.231+0000 7fee1ca36700 1 -- 172.21.15.49:0/2915485273 >> 172.21.15.49:0/2915485273 conn(0x7fee18100400 msgr2=0x7fee18101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:01.690 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.231+0000 7fee1ca36700 1 -- 172.21.15.49:0/2915485273 shutdown_connections 2023-07-20T17:43:01.690 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2023-07-20T17:43:01.232+0000 7fee1ca36700 1 -- 172.21.15.49:0/2915485273 wait complete. 2023-07-20T17:43:01.690 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-20T17:43:01.690 INFO:teuthology.orchestra.run.smithi049.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-20T17:43:01.690 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:43:01.690 INFO:teuthology.orchestra.run.smithi049.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:01.691 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:43:01.691 INFO:teuthology.orchestra.run.smithi049.stdout:Or, if you are only running a single cluster on this host: 2023-07-20T17:43:01.691 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:43:01.691 INFO:teuthology.orchestra.run.smithi049.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-20T17:43:01.691 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:43:01.691 INFO:teuthology.orchestra.run.smithi049.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-20T17:43:01.691 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:43:01.692 INFO:teuthology.orchestra.run.smithi049.stdout: ceph telemetry on 2023-07-20T17:43:01.692 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:43:01.692 INFO:teuthology.orchestra.run.smithi049.stdout:For more information see: 2023-07-20T17:43:01.692 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:43:01.692 INFO:teuthology.orchestra.run.smithi049.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-20T17:43:01.692 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:43:01.692 INFO:teuthology.orchestra.run.smithi049.stdout:Bootstrap complete. 2023-07-20T17:43:01.716 INFO:tasks.cephadm:Fetching config... 2023-07-20T17:43:01.716 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:43:01.717 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-20T17:43:01.734 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-20T17:43:01.734 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:43:01.735 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-20T17:43:01.791 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-20T17:43:01.791 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:43:01.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-20T17:43:01.860 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-20T17:43:01.861 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:43:01.861 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-20T17:43:01.918 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-20T17:43:01.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwy9yG54fHOppHe4rzNcXl5vyFil+sl+m/Cp2xfHbxUuy6+fD74k5b2iphUpGb8GoGbo9hgZRqfjNX2d+PNeDr089Cu8QaPB7oglDss1RUGlWnf0EXTLU0zuT0DwlZd24j6sDgy1ji1aIKt7vwwEXT+/zEo45GKAPoByoEHUO3f8CNg4XXgRjNaEestLxeVSlVyvDoruC45na60uKVBctm+L+b52AXMKMqivYPgbr9WRUlFQ0y3pB2rnGNN4qlUoS8Eyk9Y9gtBQoS8uE7sGtN3Safuj9M5hosHoDtnKHOoY4lq+EcbLhcR0sf8TrgSJ3zyX2m0KzOHrvTNCajBWi9a1K+RMX+C4shuHr7BJ9l7Z/smBN14wovhli+bFIQcPasEk2LGaPr3gigKSDfxEDSPSUnUVilZwRtgqtA94VwTlpE4WYX2BaR8jTIP9hbYYH0tiWLbiM7f9datpUzzjqm79eESwochrdmWf+3iasZbWEqKA6AKCOR4pCy7jb+jV0= ceph-8c050cea-2724-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T17:43:01.997 INFO:teuthology.orchestra.run.smithi049.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwy9yG54fHOppHe4rzNcXl5vyFil+sl+m/Cp2xfHbxUuy6+fD74k5b2iphUpGb8GoGbo9hgZRqfjNX2d+PNeDr089Cu8QaPB7oglDss1RUGlWnf0EXTLU0zuT0DwlZd24j6sDgy1ji1aIKt7vwwEXT+/zEo45GKAPoByoEHUO3f8CNg4XXgRjNaEestLxeVSlVyvDoruC45na60uKVBctm+L+b52AXMKMqivYPgbr9WRUlFQ0y3pB2rnGNN4qlUoS8Eyk9Y9gtBQoS8uE7sGtN3Safuj9M5hosHoDtnKHOoY4lq+EcbLhcR0sf8TrgSJ3zyX2m0KzOHrvTNCajBWi9a1K+RMX+C4shuHr7BJ9l7Z/smBN14wovhli+bFIQcPasEk2LGaPr3gigKSDfxEDSPSUnUVilZwRtgqtA94VwTlpE4WYX2BaR8jTIP9hbYYH0tiWLbiM7f9datpUzzjqm79eESwochrdmWf+3iasZbWEqKA6AKCOR4pCy7jb+jV0= ceph-8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:43:02.009 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwy9yG54fHOppHe4rzNcXl5vyFil+sl+m/Cp2xfHbxUuy6+fD74k5b2iphUpGb8GoGbo9hgZRqfjNX2d+PNeDr089Cu8QaPB7oglDss1RUGlWnf0EXTLU0zuT0DwlZd24j6sDgy1ji1aIKt7vwwEXT+/zEo45GKAPoByoEHUO3f8CNg4XXgRjNaEestLxeVSlVyvDoruC45na60uKVBctm+L+b52AXMKMqivYPgbr9WRUlFQ0y3pB2rnGNN4qlUoS8Eyk9Y9gtBQoS8uE7sGtN3Safuj9M5hosHoDtnKHOoY4lq+EcbLhcR0sf8TrgSJ3zyX2m0KzOHrvTNCajBWi9a1K+RMX+C4shuHr7BJ9l7Z/smBN14wovhli+bFIQcPasEk2LGaPr3gigKSDfxEDSPSUnUVilZwRtgqtA94VwTlpE4WYX2BaR8jTIP9hbYYH0tiWLbiM7f9datpUzzjqm79eESwochrdmWf+3iasZbWEqKA6AKCOR4pCy7jb+jV0= ceph-8c050cea-2724-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T17:43:02.053 INFO:teuthology.orchestra.run.smithi079.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwy9yG54fHOppHe4rzNcXl5vyFil+sl+m/Cp2xfHbxUuy6+fD74k5b2iphUpGb8GoGbo9hgZRqfjNX2d+PNeDr089Cu8QaPB7oglDss1RUGlWnf0EXTLU0zuT0DwlZd24j6sDgy1ji1aIKt7vwwEXT+/zEo45GKAPoByoEHUO3f8CNg4XXgRjNaEestLxeVSlVyvDoruC45na60uKVBctm+L+b52AXMKMqivYPgbr9WRUlFQ0y3pB2rnGNN4qlUoS8Eyk9Y9gtBQoS8uE7sGtN3Safuj9M5hosHoDtnKHOoY4lq+EcbLhcR0sf8TrgSJ3zyX2m0KzOHrvTNCajBWi9a1K+RMX+C4shuHr7BJ9l7Z/smBN14wovhli+bFIQcPasEk2LGaPr3gigKSDfxEDSPSUnUVilZwRtgqtA94VwTlpE4WYX2BaR8jTIP9hbYYH0tiWLbiM7f9datpUzzjqm79eESwochrdmWf+3iasZbWEqKA6AKCOR4pCy7jb+jV0= ceph-8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:43:02.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwy9yG54fHOppHe4rzNcXl5vyFil+sl+m/Cp2xfHbxUuy6+fD74k5b2iphUpGb8GoGbo9hgZRqfjNX2d+PNeDr089Cu8QaPB7oglDss1RUGlWnf0EXTLU0zuT0DwlZd24j6sDgy1ji1aIKt7vwwEXT+/zEo45GKAPoByoEHUO3f8CNg4XXgRjNaEestLxeVSlVyvDoruC45na60uKVBctm+L+b52AXMKMqivYPgbr9WRUlFQ0y3pB2rnGNN4qlUoS8Eyk9Y9gtBQoS8uE7sGtN3Safuj9M5hosHoDtnKHOoY4lq+EcbLhcR0sf8TrgSJ3zyX2m0KzOHrvTNCajBWi9a1K+RMX+C4shuHr7BJ9l7Z/smBN14wovhli+bFIQcPasEk2LGaPr3gigKSDfxEDSPSUnUVilZwRtgqtA94VwTlpE4WYX2BaR8jTIP9hbYYH0tiWLbiM7f9datpUzzjqm79eESwochrdmWf+3iasZbWEqKA6AKCOR4pCy7jb+jV0= ceph-8c050cea-2724-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T17:43:02.111 INFO:teuthology.orchestra.run.smithi097.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwy9yG54fHOppHe4rzNcXl5vyFil+sl+m/Cp2xfHbxUuy6+fD74k5b2iphUpGb8GoGbo9hgZRqfjNX2d+PNeDr089Cu8QaPB7oglDss1RUGlWnf0EXTLU0zuT0DwlZd24j6sDgy1ji1aIKt7vwwEXT+/zEo45GKAPoByoEHUO3f8CNg4XXgRjNaEestLxeVSlVyvDoruC45na60uKVBctm+L+b52AXMKMqivYPgbr9WRUlFQ0y3pB2rnGNN4qlUoS8Eyk9Y9gtBQoS8uE7sGtN3Safuj9M5hosHoDtnKHOoY4lq+EcbLhcR0sf8TrgSJ3zyX2m0KzOHrvTNCajBWi9a1K+RMX+C4shuHr7BJ9l7Z/smBN14wovhli+bFIQcPasEk2LGaPr3gigKSDfxEDSPSUnUVilZwRtgqtA94VwTlpE4WYX2BaR8jTIP9hbYYH0tiWLbiM7f9datpUzzjqm79eESwochrdmWf+3iasZbWEqKA6AKCOR4pCy7jb+jV0= ceph-8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:43:02.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-20T17:43:02.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:02 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2915485273' entity='client.admin' 2023-07-20T17:43:02.688 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:43:04.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.284+0000 7f0f66101700 1 -- 172.21.15.49:0/1893381444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f600ff160 msgr2=0x7f0f600ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:04.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.284+0000 7f0f66101700 1 --2- 172.21.15.49:0/1893381444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f600ff160 0x7f0f600ff580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009910 tx=0x7f0f50009c20 comp rx=0 tx=0).stop 2023-07-20T17:43:04.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.284+0000 7f0f66101700 1 -- 172.21.15.49:0/1893381444 shutdown_connections 2023-07-20T17:43:04.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.284+0000 7f0f66101700 1 --2- 172.21.15.49:0/1893381444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f600ff160 0x7f0f600ff580 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:04.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.284+0000 7f0f66101700 1 -- 172.21.15.49:0/1893381444 >> 172.21.15.49:0/1893381444 conn(0x7f0f600fa700 msgr2=0x7f0f600fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:04.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.284+0000 7f0f66101700 1 -- 172.21.15.49:0/1893381444 shutdown_connections 2023-07-20T17:43:04.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.284+0000 7f0f66101700 1 -- 172.21.15.49:0/1893381444 wait complete. 2023-07-20T17:43:04.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.285+0000 7f0f66101700 1 Processor -- start 2023-07-20T17:43:04.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.285+0000 7f0f66101700 1 -- start start 2023-07-20T17:43:04.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.285+0000 7f0f66101700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f600ff160 0x7f0f600684d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:04.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.285+0000 7f0f66101700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f601980c0 con 0x7f0f600ff160 2023-07-20T17:43:04.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.285+0000 7f0f5f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f600ff160 0x7f0f600684d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:04.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.285+0000 7f0f5f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f600ff160 0x7f0f600684d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35478/0 (socket says 172.21.15.49:35478) 2023-07-20T17:43:04.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.285+0000 7f0f5f7fe700 1 -- 172.21.15.49:0/1546314334 learned_addr learned my addr 172.21.15.49:0/1546314334 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:04.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.286+0000 7f0f5f7fe700 1 -- 172.21.15.49:0/1546314334 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f500095f0 con 0x7f0f600ff160 2023-07-20T17:43:04.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.286+0000 7f0f5f7fe700 1 --2- 172.21.15.49:0/1546314334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f600ff160 0x7f0f600684d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0f50006010 tx=0x7f0f500057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:04.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.286+0000 7f0f5cff9700 1 -- 172.21.15.49:0/1546314334 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f5001d070 con 0x7f0f600ff160 2023-07-20T17:43:04.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.286+0000 7f0f66101700 1 -- 172.21.15.49:0/1546314334 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f60198200 con 0x7f0f600ff160 2023-07-20T17:43:04.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.286+0000 7f0f5cff9700 1 -- 172.21.15.49:0/1546314334 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f500044e0 con 0x7f0f600ff160 2023-07-20T17:43:04.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.286+0000 7f0f5cff9700 1 -- 172.21.15.49:0/1546314334 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f500039b0 con 0x7f0f600ff160 2023-07-20T17:43:04.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.286+0000 7f0f66101700 1 -- 172.21.15.49:0/1546314334 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f60198640 con 0x7f0f600ff160 2023-07-20T17:43:04.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.287+0000 7f0f5cff9700 1 -- 172.21.15.49:0/1546314334 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f0f50003c70 con 0x7f0f600ff160 2023-07-20T17:43:04.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.287+0000 7f0f66101700 1 -- 172.21.15.49:0/1546314334 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6018dc70 con 0x7f0f600ff160 2023-07-20T17:43:04.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.287+0000 7f0f5cff9700 1 --2- 172.21.15.49:0/1546314334 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f480387b0 0x7f0f4803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:04.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.287+0000 7f0f5cff9700 1 -- 172.21.15.49:0/1546314334 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0f5004ce30 con 0x7f0f600ff160 2023-07-20T17:43:04.297 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.287+0000 7f0f5effd700 1 --2- 172.21.15.49:0/1546314334 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f480387b0 0x7f0f4803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:04.297 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.287+0000 7f0f5effd700 1 --2- 172.21.15.49:0/1546314334 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f480387b0 0x7f0f4803ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0f54006fd0 tx=0x7f0f54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:04.297 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.293+0000 7f0f5cff9700 1 -- 172.21.15.49:0/1546314334 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0f5002a950 con 0x7f0f600ff160 2023-07-20T17:43:04.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:04 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:04.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:04 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:04.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:04 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi049", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:43:04.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:04 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:04.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:04 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:04.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:04 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:04.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.492+0000 7f0f66101700 1 -- 172.21.15.49:0/1546314334 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f0f6002ca60 con 0x7f0f600ff160 2023-07-20T17:43:04.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.495+0000 7f0f5cff9700 1 -- 172.21.15.49:0/1546314334 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0f5002a360 con 0x7f0f600ff160 2023-07-20T17:43:04.498 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.498+0000 7f0f66101700 1 -- 172.21.15.49:0/1546314334 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f480387b0 msgr2=0x7f0f4803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:04.498 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.498+0000 7f0f66101700 1 --2- 172.21.15.49:0/1546314334 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f480387b0 0x7f0f4803ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0f54006fd0 tx=0x7f0f54006e40 comp rx=0 tx=0).stop 2023-07-20T17:43:04.498 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.498+0000 7f0f66101700 1 -- 172.21.15.49:0/1546314334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f600ff160 msgr2=0x7f0f600684d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:04.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.498+0000 7f0f66101700 1 --2- 172.21.15.49:0/1546314334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f600ff160 0x7f0f600684d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0f50006010 tx=0x7f0f500057c0 comp rx=0 tx=0).stop 2023-07-20T17:43:04.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.498+0000 7f0f66101700 1 -- 172.21.15.49:0/1546314334 shutdown_connections 2023-07-20T17:43:04.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.498+0000 7f0f66101700 1 --2- 172.21.15.49:0/1546314334 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f480387b0 0x7f0f4803ac70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:04.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.498+0000 7f0f66101700 1 --2- 172.21.15.49:0/1546314334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f600ff160 0x7f0f600684d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:04.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.498+0000 7f0f66101700 1 -- 172.21.15.49:0/1546314334 >> 172.21.15.49:0/1546314334 conn(0x7f0f600fa700 msgr2=0x7f0f600fb480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:04.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.498+0000 7f0f66101700 1 -- 172.21.15.49:0/1546314334 shutdown_connections 2023-07-20T17:43:04.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:04.498+0000 7f0f66101700 1 -- 172.21.15.49:0/1546314334 wait complete. 2023-07-20T17:43:04.845 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-20T17:43:04.846 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-20T17:43:05.402 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:43:05.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:05 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1546314334' entity='client.admin' 2023-07-20T17:43:05.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:05.818 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.618+0000 7f2814417700 1 -- 172.21.15.49:0/522936401 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f280c0ff160 msgr2=0x7f280c0ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:06.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.618+0000 7f2814417700 1 --2- 172.21.15.49:0/522936401 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f280c0ff160 0x7f280c0ff580 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009910 tx=0x7f27fc009c20 comp rx=0 tx=0).stop 2023-07-20T17:43:06.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.618+0000 7f2814417700 1 -- 172.21.15.49:0/522936401 shutdown_connections 2023-07-20T17:43:06.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.618+0000 7f2814417700 1 --2- 172.21.15.49:0/522936401 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f280c0ff160 0x7f280c0ff580 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:06.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.618+0000 7f2814417700 1 -- 172.21.15.49:0/522936401 >> 172.21.15.49:0/522936401 conn(0x7f280c0fa700 msgr2=0x7f280c0fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:06.620 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.618+0000 7f2814417700 1 -- 172.21.15.49:0/522936401 shutdown_connections 2023-07-20T17:43:06.620 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.618+0000 7f2814417700 1 -- 172.21.15.49:0/522936401 wait complete. 2023-07-20T17:43:06.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.619+0000 7f2814417700 1 Processor -- start 2023-07-20T17:43:06.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.619+0000 7f2814417700 1 -- start start 2023-07-20T17:43:06.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.619+0000 7f2814417700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f280c0ff160 0x7f280c194270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:06.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.619+0000 7f2814417700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280c1947b0 con 0x7f280c0ff160 2023-07-20T17:43:06.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.619+0000 7f28121b3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f280c0ff160 0x7f280c194270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:06.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.619+0000 7f28121b3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f280c0ff160 0x7f280c194270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35484/0 (socket says 172.21.15.49:35484) 2023-07-20T17:43:06.623 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.619+0000 7f28121b3700 1 -- 172.21.15.49:0/400809757 learned_addr learned my addr 172.21.15.49:0/400809757 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:06.623 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.619+0000 7f28121b3700 1 -- 172.21.15.49:0/400809757 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc0095f0 con 0x7f280c0ff160 2023-07-20T17:43:06.623 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.619+0000 7f28121b3700 1 --2- 172.21.15.49:0/400809757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f280c0ff160 0x7f280c194270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f27fc004f40 tx=0x7f27fc0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:06.623 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.619+0000 7f28037fe700 1 -- 172.21.15.49:0/400809757 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27fc01d070 con 0x7f280c0ff160 2023-07-20T17:43:06.623 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.619+0000 7f28037fe700 1 -- 172.21.15.49:0/400809757 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27fc00bc10 con 0x7f280c0ff160 2023-07-20T17:43:06.623 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.620+0000 7f2814417700 1 -- 172.21.15.49:0/400809757 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280c1949b0 con 0x7f280c0ff160 2023-07-20T17:43:06.623 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.620+0000 7f28037fe700 1 -- 172.21.15.49:0/400809757 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27fc003a70 con 0x7f280c0ff160 2023-07-20T17:43:06.624 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.620+0000 7f2814417700 1 -- 172.21.15.49:0/400809757 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f280c194e50 con 0x7f280c0ff160 2023-07-20T17:43:06.624 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.620+0000 7f28037fe700 1 -- 172.21.15.49:0/400809757 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f27fc003d30 con 0x7f280c0ff160 2023-07-20T17:43:06.624 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.620+0000 7f2814417700 1 -- 172.21.15.49:0/400809757 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280c18dc70 con 0x7f280c0ff160 2023-07-20T17:43:06.624 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.621+0000 7f28037fe700 1 --2- 172.21.15.49:0/400809757 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f27f80387b0 0x7f27f803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:06.624 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.621+0000 7f28037fe700 1 -- 172.21.15.49:0/400809757 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f27fc04c4e0 con 0x7f280c0ff160 2023-07-20T17:43:06.624 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.621+0000 7f28119b2700 1 --2- 172.21.15.49:0/400809757 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f27f80387b0 0x7f27f803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:06.625 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.621+0000 7f28119b2700 1 --2- 172.21.15.49:0/400809757 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f27f80387b0 0x7f27f803ac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2808006fd0 tx=0x7f2808006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:06.625 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.625+0000 7f28037fe700 1 -- 172.21.15.49:0/400809757 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f27fc02ae20 con 0x7f280c0ff160 2023-07-20T17:43:06.826 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.826+0000 7f2814417700 1 -- 172.21.15.49:0/400809757 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f280c060c40 con 0x7f27f80387b0 2023-07-20T17:43:06.829 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.829+0000 7f28037fe700 1 -- 172.21.15.49:0/400809757 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f280c060c40 con 0x7f27f80387b0 2023-07-20T17:43:06.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.831+0000 7f2814417700 1 -- 172.21.15.49:0/400809757 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f27f80387b0 msgr2=0x7f27f803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:06.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.831+0000 7f2814417700 1 --2- 172.21.15.49:0/400809757 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f27f80387b0 0x7f27f803ac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2808006fd0 tx=0x7f2808006e40 comp rx=0 tx=0).stop 2023-07-20T17:43:06.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.831+0000 7f2814417700 1 -- 172.21.15.49:0/400809757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f280c0ff160 msgr2=0x7f280c194270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:06.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.831+0000 7f2814417700 1 --2- 172.21.15.49:0/400809757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f280c0ff160 0x7f280c194270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f27fc004f40 tx=0x7f27fc0057c0 comp rx=0 tx=0).stop 2023-07-20T17:43:06.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.831+0000 7f2814417700 1 -- 172.21.15.49:0/400809757 shutdown_connections 2023-07-20T17:43:06.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.831+0000 7f2814417700 1 --2- 172.21.15.49:0/400809757 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f27f80387b0 0x7f27f803ac70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:06.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.831+0000 7f2814417700 1 --2- 172.21.15.49:0/400809757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f280c0ff160 0x7f280c194270 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:06.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.831+0000 7f2814417700 1 -- 172.21.15.49:0/400809757 >> 172.21.15.49:0/400809757 conn(0x7f280c0fa700 msgr2=0x7f280c0fb480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:06.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.831+0000 7f2814417700 1 -- 172.21.15.49:0/400809757 shutdown_connections 2023-07-20T17:43:06.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:06.831+0000 7f2814417700 1 -- 172.21.15.49:0/400809757 wait complete. 2023-07-20T17:43:07.261 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi079 2023-07-20T17:43:07.261 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-07-20T17:43:07.261 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/etc/ceph/ceph.conf 2023-07-20T17:43:07.282 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-07-20T17:43:07.283 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:07.341 INFO:tasks.cephadm:Adding host smithi079 to orchestrator... 2023-07-20T17:43:07.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch host add smithi079 2023-07-20T17:43:07.907 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:43:08.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:07 smithi049 ceph-mon[96530]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:08.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:07 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:08.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:07 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:08.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:07 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:08.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:07 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:08.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:07 smithi049 ceph-mon[96530]: Updating smithi049:/etc/ceph/ceph.conf 2023-07-20T17:43:08.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:07 smithi049 ceph-mon[96530]: Updating smithi049:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:08.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:07 smithi049 ceph-mon[96530]: Updating smithi049:/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:08.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:07 smithi049 ceph-mon[96530]: Updating smithi049:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T17:43:09.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.215+0000 7fc1d424a700 1 -- 172.21.15.49:0/2779641693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1cc100450 msgr2=0x7fc1cc100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:09.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.215+0000 7fc1d424a700 1 --2- 172.21.15.49:0/2779641693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1cc100450 0x7fc1cc100830 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009910 tx=0x7fc1bc009c20 comp rx=0 tx=0).stop 2023-07-20T17:43:09.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.215+0000 7fc1d424a700 1 -- 172.21.15.49:0/2779641693 shutdown_connections 2023-07-20T17:43:09.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.215+0000 7fc1d424a700 1 --2- 172.21.15.49:0/2779641693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1cc100450 0x7fc1cc100830 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:09.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.215+0000 7fc1d424a700 1 -- 172.21.15.49:0/2779641693 >> 172.21.15.49:0/2779641693 conn(0x7fc1cc0fbd00 msgr2=0x7fc1cc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:09.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.215+0000 7fc1d424a700 1 -- 172.21.15.49:0/2779641693 shutdown_connections 2023-07-20T17:43:09.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.215+0000 7fc1d424a700 1 -- 172.21.15.49:0/2779641693 wait complete. 2023-07-20T17:43:09.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.216+0000 7fc1d424a700 1 Processor -- start 2023-07-20T17:43:09.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.216+0000 7fc1d424a700 1 -- start start 2023-07-20T17:43:09.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.216+0000 7fc1d424a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1cc100450 0x7fc1cc1956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:09.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.216+0000 7fc1d424a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc195c00 con 0x7fc1cc100450 2023-07-20T17:43:09.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.217+0000 7fc1d1fe6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1cc100450 0x7fc1cc1956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:09.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.217+0000 7fc1d1fe6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1cc100450 0x7fc1cc1956c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35490/0 (socket says 172.21.15.49:35490) 2023-07-20T17:43:09.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.217+0000 7fc1d1fe6700 1 -- 172.21.15.49:0/4254530313 learned_addr learned my addr 172.21.15.49:0/4254530313 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:09.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.217+0000 7fc1d1fe6700 1 -- 172.21.15.49:0/4254530313 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc0095f0 con 0x7fc1cc100450 2023-07-20T17:43:09.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.217+0000 7fc1d1fe6700 1 --2- 172.21.15.49:0/4254530313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1cc100450 0x7fc1cc1956c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc004f40 tx=0x7fc1bc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:09.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.217+0000 7fc1c2ffd700 1 -- 172.21.15.49:0/4254530313 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1bc01d070 con 0x7fc1cc100450 2023-07-20T17:43:09.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.217+0000 7fc1d424a700 1 -- 172.21.15.49:0/4254530313 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc195e00 con 0x7fc1cc100450 2023-07-20T17:43:09.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.217+0000 7fc1c2ffd700 1 -- 172.21.15.49:0/4254530313 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1bc004370 con 0x7fc1cc100450 2023-07-20T17:43:09.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.217+0000 7fc1c2ffd700 1 -- 172.21.15.49:0/4254530313 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1bc0039b0 con 0x7fc1cc100450 2023-07-20T17:43:09.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.217+0000 7fc1d424a700 1 -- 172.21.15.49:0/4254530313 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc196220 con 0x7fc1cc100450 2023-07-20T17:43:09.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.218+0000 7fc1c2ffd700 1 -- 172.21.15.49:0/4254530313 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fc1bc003c70 con 0x7fc1cc100450 2023-07-20T17:43:09.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.218+0000 7fc1c2ffd700 1 --2- 172.21.15.49:0/4254530313 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc1b80387b0 0x7fc1b803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:09.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.218+0000 7fc1c2ffd700 1 -- 172.21.15.49:0/4254530313 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc1bc04d4d0 con 0x7fc1cc100450 2023-07-20T17:43:09.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.218+0000 7fc1d17e5700 1 --2- 172.21.15.49:0/4254530313 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc1b80387b0 0x7fc1b803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:09.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.218+0000 7fc1d424a700 1 -- 172.21.15.49:0/4254530313 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b0005320 con 0x7fc1cc100450 2023-07-20T17:43:09.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.219+0000 7fc1d17e5700 1 --2- 172.21.15.49:0/4254530313 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc1b80387b0 0x7fc1b803ac70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8006fd0 tx=0x7fc1c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:09.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.223+0000 7fc1c2ffd700 1 -- 172.21.15.49:0/4254530313 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc1bc026020 con 0x7fc1cc100450 2023-07-20T17:43:09.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:09.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:09.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:09.423+0000 7fc1d424a700 1 -- 172.21.15.49:0/4254530313 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi079", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1b0000bf0 con 0x7fc1b80387b0 2023-07-20T17:43:10.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:09 smithi049 ceph-mon[96530]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi079", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:11.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:10 smithi049 ceph-mon[96530]: Deploying cephadm binary to smithi079 2023-07-20T17:43:12.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:12.202+0000 7fc1c2ffd700 1 -- 172.21.15.49:0/4254530313 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fc1b0000bf0 con 0x7fc1b80387b0 2023-07-20T17:43:12.203 INFO:teuthology.orchestra.run.smithi049.stdout:Added host 'smithi079' with addr '172.21.15.79' 2023-07-20T17:43:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:12.204+0000 7fc1d424a700 1 -- 172.21.15.49:0/4254530313 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc1b80387b0 msgr2=0x7fc1b803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:12.204+0000 7fc1d424a700 1 --2- 172.21.15.49:0/4254530313 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc1b80387b0 0x7fc1b803ac70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8006fd0 tx=0x7fc1c8006e40 comp rx=0 tx=0).stop 2023-07-20T17:43:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:12.204+0000 7fc1d424a700 1 -- 172.21.15.49:0/4254530313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1cc100450 msgr2=0x7fc1cc1956c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:12.204+0000 7fc1d424a700 1 --2- 172.21.15.49:0/4254530313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1cc100450 0x7fc1cc1956c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc004f40 tx=0x7fc1bc005020 comp rx=0 tx=0).stop 2023-07-20T17:43:12.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:12.204+0000 7fc1d424a700 1 -- 172.21.15.49:0/4254530313 shutdown_connections 2023-07-20T17:43:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:12.204+0000 7fc1d424a700 1 --2- 172.21.15.49:0/4254530313 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc1b80387b0 0x7fc1b803ac70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:12.204+0000 7fc1d424a700 1 --2- 172.21.15.49:0/4254530313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1cc100450 0x7fc1cc1956c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:12.204+0000 7fc1d424a700 1 -- 172.21.15.49:0/4254530313 >> 172.21.15.49:0/4254530313 conn(0x7fc1cc0fbd00 msgr2=0x7fc1cc1046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:12.204+0000 7fc1d424a700 1 -- 172.21.15.49:0/4254530313 shutdown_connections 2023-07-20T17:43:12.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:12.204+0000 7fc1d424a700 1 -- 172.21.15.49:0/4254530313 wait complete. 2023-07-20T17:43:12.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T17:43:13.235 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:43:13.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:13 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:13.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:13 smithi049 ceph-mon[96530]: Added host smithi079 2023-07-20T17:43:13.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:13 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:13.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:13 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:13.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:13 smithi049 ceph-mon[96530]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:14.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.446+0000 7f0f77184700 1 -- 172.21.15.49:0/3859805538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70105180 msgr2=0x7f0f70105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:14.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.446+0000 7f0f77184700 1 --2- 172.21.15.49:0/3859805538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70105180 0x7f0f70105560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009910 tx=0x7f0f64009c20 comp rx=0 tx=0).stop 2023-07-20T17:43:14.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.446+0000 7f0f77184700 1 -- 172.21.15.49:0/3859805538 shutdown_connections 2023-07-20T17:43:14.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.446+0000 7f0f77184700 1 --2- 172.21.15.49:0/3859805538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70105180 0x7f0f70105560 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:14.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.446+0000 7f0f77184700 1 -- 172.21.15.49:0/3859805538 >> 172.21.15.49:0/3859805538 conn(0x7f0f70074c70 msgr2=0x7f0f70075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:14.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.446+0000 7f0f77184700 1 -- 172.21.15.49:0/3859805538 shutdown_connections 2023-07-20T17:43:14.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.446+0000 7f0f77184700 1 -- 172.21.15.49:0/3859805538 wait complete. 2023-07-20T17:43:14.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.446+0000 7f0f77184700 1 Processor -- start 2023-07-20T17:43:14.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.447+0000 7f0f77184700 1 -- start start 2023-07-20T17:43:14.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.447+0000 7f0f77184700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70105180 0x7f0f7019d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:14.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.447+0000 7f0f77184700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7019dba0 con 0x7f0f70105180 2023-07-20T17:43:14.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.447+0000 7f0f74f20700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70105180 0x7f0f7019d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:14.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.447+0000 7f0f74f20700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70105180 0x7f0f7019d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35516/0 (socket says 172.21.15.49:35516) 2023-07-20T17:43:14.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.447+0000 7f0f74f20700 1 -- 172.21.15.49:0/3997382566 learned_addr learned my addr 172.21.15.49:0/3997382566 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:14.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.447+0000 7f0f74f20700 1 -- 172.21.15.49:0/3997382566 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f640095f0 con 0x7f0f70105180 2023-07-20T17:43:14.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.447+0000 7f0f74f20700 1 --2- 172.21.15.49:0/3997382566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70105180 0x7f0f7019d660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0f64004f40 tx=0x7f0f64005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:14.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.447+0000 7f0f6dffb700 1 -- 172.21.15.49:0/3997382566 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f6401d070 con 0x7f0f70105180 2023-07-20T17:43:14.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.447+0000 7f0f6dffb700 1 -- 172.21.15.49:0/3997382566 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f64004370 con 0x7f0f70105180 2023-07-20T17:43:14.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.447+0000 7f0f77184700 1 -- 172.21.15.49:0/3997382566 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7019dda0 con 0x7f0f70105180 2023-07-20T17:43:14.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.448+0000 7f0f6dffb700 1 -- 172.21.15.49:0/3997382566 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f64003970 con 0x7f0f70105180 2023-07-20T17:43:14.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.448+0000 7f0f77184700 1 -- 172.21.15.49:0/3997382566 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f701976e0 con 0x7f0f70105180 2023-07-20T17:43:14.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.448+0000 7f0f6dffb700 1 -- 172.21.15.49:0/3997382566 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f0f64003c30 con 0x7f0f70105180 2023-07-20T17:43:14.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.449+0000 7f0f6dffb700 1 --2- 172.21.15.49:0/3997382566 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f600387a0 0x7f0f6003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:14.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.449+0000 7f0f77184700 1 -- 172.21.15.49:0/3997382566 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f70061da0 con 0x7f0f70105180 2023-07-20T17:43:14.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.449+0000 7f0f6dffb700 1 -- 172.21.15.49:0/3997382566 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0f6404c4d0 con 0x7f0f70105180 2023-07-20T17:43:14.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.449+0000 7f0f6ffff700 1 --2- 172.21.15.49:0/3997382566 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f600387a0 0x7f0f6003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:14.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.449+0000 7f0f6ffff700 1 --2- 172.21.15.49:0/3997382566 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f600387a0 0x7f0f6003ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c006fd0 tx=0x7f0f5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:14.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.454+0000 7f0f6dffb700 1 -- 172.21.15.49:0/3997382566 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0f6402a590 con 0x7f0f70105180 2023-07-20T17:43:14.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.663+0000 7f0f77184700 1 -- 172.21.15.49:0/3997382566 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f70197920 con 0x7f0f600387a0 2023-07-20T17:43:14.665 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.664+0000 7f0f6dffb700 1 -- 172.21.15.49:0/3997382566 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f0f70197920 con 0x7f0f600387a0 2023-07-20T17:43:14.665 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:43:14.665 INFO:teuthology.orchestra.run.smithi049.stdout:[{"addr": "172.21.15.49", "hostname": "smithi049", "labels": [], "status": ""}, {"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}] 2023-07-20T17:43:14.667 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.666+0000 7f0f77184700 1 -- 172.21.15.49:0/3997382566 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f600387a0 msgr2=0x7f0f6003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:14.667 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.666+0000 7f0f77184700 1 --2- 172.21.15.49:0/3997382566 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f600387a0 0x7f0f6003ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c006fd0 tx=0x7f0f5c006e40 comp rx=0 tx=0).stop 2023-07-20T17:43:14.667 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.666+0000 7f0f77184700 1 -- 172.21.15.49:0/3997382566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70105180 msgr2=0x7f0f7019d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:14.667 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.666+0000 7f0f77184700 1 --2- 172.21.15.49:0/3997382566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70105180 0x7f0f7019d660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0f64004f40 tx=0x7f0f64005020 comp rx=0 tx=0).stop 2023-07-20T17:43:14.668 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.667+0000 7f0f77184700 1 -- 172.21.15.49:0/3997382566 shutdown_connections 2023-07-20T17:43:14.668 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.667+0000 7f0f77184700 1 --2- 172.21.15.49:0/3997382566 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0f600387a0 0x7f0f6003ac60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:14.668 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.667+0000 7f0f77184700 1 --2- 172.21.15.49:0/3997382566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70105180 0x7f0f7019d660 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:14.668 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.667+0000 7f0f77184700 1 -- 172.21.15.49:0/3997382566 >> 172.21.15.49:0/3997382566 conn(0x7f0f70074c70 msgr2=0x7f0f700fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:14.668 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.667+0000 7f0f77184700 1 -- 172.21.15.49:0/3997382566 shutdown_connections 2023-07-20T17:43:14.668 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:14.667+0000 7f0f77184700 1 -- 172.21.15.49:0/3997382566 wait complete. 2023-07-20T17:43:14.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:14 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:15.091 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi097 2023-07-20T17:43:15.092 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-07-20T17:43:15.092 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/etc/ceph/ceph.conf 2023-07-20T17:43:15.112 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-07-20T17:43:15.112 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:15.172 INFO:tasks.cephadm:Adding host smithi097 to orchestrator... 2023-07-20T17:43:15.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch host add smithi097 2023-07-20T17:43:15.730 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:43:15.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:15 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:15.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:15 smithi049 ceph-mon[96530]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T17:43:15.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:15 smithi049 ceph-mon[96530]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:16.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.921+0000 7fd09bf36700 1 -- 172.21.15.49:0/3944399025 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd094104120 msgr2=0x7fd094104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:16.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.921+0000 7fd09bf36700 1 --2- 172.21.15.49:0/3944399025 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd094104120 0x7fd094104500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd090009910 tx=0x7fd090009c20 comp rx=0 tx=0).stop 2023-07-20T17:43:16.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.921+0000 7fd09bf36700 1 -- 172.21.15.49:0/3944399025 shutdown_connections 2023-07-20T17:43:16.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.921+0000 7fd09bf36700 1 --2- 172.21.15.49:0/3944399025 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd094104120 0x7fd094104500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:16.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.921+0000 7fd09bf36700 1 -- 172.21.15.49:0/3944399025 >> 172.21.15.49:0/3944399025 conn(0x7fd0940757d0 msgr2=0x7fd094075be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:16.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.921+0000 7fd09bf36700 1 -- 172.21.15.49:0/3944399025 shutdown_connections 2023-07-20T17:43:16.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.921+0000 7fd09bf36700 1 -- 172.21.15.49:0/3944399025 wait complete. 2023-07-20T17:43:16.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.922+0000 7fd09bf36700 1 Processor -- start 2023-07-20T17:43:16.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.922+0000 7fd09bf36700 1 -- start start 2023-07-20T17:43:16.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.922+0000 7fd09bf36700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd094104120 0x7fd0940fb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:16.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.922+0000 7fd09bf36700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0940fd730 con 0x7fd094104120 2023-07-20T17:43:16.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.922+0000 7fd099cd2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd094104120 0x7fd0940fb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:16.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.922+0000 7fd099cd2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd094104120 0x7fd0940fb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35522/0 (socket says 172.21.15.49:35522) 2023-07-20T17:43:16.925 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.922+0000 7fd099cd2700 1 -- 172.21.15.49:0/1754107789 learned_addr learned my addr 172.21.15.49:0/1754107789 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:16.925 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.922+0000 7fd099cd2700 1 -- 172.21.15.49:0/1754107789 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0900095f0 con 0x7fd094104120 2023-07-20T17:43:16.926 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.923+0000 7fd099cd2700 1 --2- 172.21.15.49:0/1754107789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd094104120 0x7fd0940fb7b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd090004f40 tx=0x7fd090005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:16.926 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.923+0000 7fd086ffd700 1 -- 172.21.15.49:0/1754107789 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd09001d070 con 0x7fd094104120 2023-07-20T17:43:16.927 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.923+0000 7fd086ffd700 1 -- 172.21.15.49:0/1754107789 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd090004370 con 0x7fd094104120 2023-07-20T17:43:16.927 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.923+0000 7fd09bf36700 1 -- 172.21.15.49:0/1754107789 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0940fbcf0 con 0x7fd094104120 2023-07-20T17:43:16.927 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.923+0000 7fd086ffd700 1 -- 172.21.15.49:0/1754107789 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0900039b0 con 0x7fd094104120 2023-07-20T17:43:16.927 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.923+0000 7fd09bf36700 1 -- 172.21.15.49:0/1754107789 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0940fc1f0 con 0x7fd094104120 2023-07-20T17:43:16.927 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.924+0000 7fd086ffd700 1 -- 172.21.15.49:0/1754107789 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fd090003c70 con 0x7fd094104120 2023-07-20T17:43:16.927 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.924+0000 7fd09bf36700 1 -- 172.21.15.49:0/1754107789 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd094061da0 con 0x7fd094104120 2023-07-20T17:43:16.928 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.924+0000 7fd086ffd700 1 --2- 172.21.15.49:0/1754107789 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd0800387b0 0x7fd08003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:16.928 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.924+0000 7fd086ffd700 1 -- 172.21.15.49:0/1754107789 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd09004c5a0 con 0x7fd094104120 2023-07-20T17:43:16.928 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.924+0000 7fd0994d1700 1 --2- 172.21.15.49:0/1754107789 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd0800387b0 0x7fd08003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:16.928 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.924+0000 7fd0994d1700 1 --2- 172.21.15.49:0/1754107789 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd0800387b0 0x7fd08003ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd088006fd0 tx=0x7fd088006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:16.929 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:16.929+0000 7fd086ffd700 1 -- 172.21.15.49:0/1754107789 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd090029360 con 0x7fd094104120 2023-07-20T17:43:17.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:17.136+0000 7fd09bf36700 1 -- 172.21.15.49:0/1754107789 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi097", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0940fc5e0 con 0x7fd0800387b0 2023-07-20T17:43:17.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:16 smithi049 ceph-mon[96530]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:18.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:18.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:18.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:18.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:18.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:43:18.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:18.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:18.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: Updating smithi079:/etc/ceph/ceph.conf 2023-07-20T17:43:18.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi097", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:18.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: Updating smithi079:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:18.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:18.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:18 smithi049 ceph-mon[96530]: Deploying cephadm binary to smithi097 2023-07-20T17:43:19.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:19 smithi049 ceph-mon[96530]: Updating smithi079:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T17:43:19.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:19.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:19.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:19.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:19 smithi049 ceph-mon[96530]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:19.942 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:19.941+0000 7fd086ffd700 1 -- 172.21.15.49:0/1754107789 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fd0940fc5e0 con 0x7fd0800387b0 2023-07-20T17:43:19.942 INFO:teuthology.orchestra.run.smithi049.stdout:Added host 'smithi097' with addr '172.21.15.97' 2023-07-20T17:43:19.944 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:19.944+0000 7fd09bf36700 1 -- 172.21.15.49:0/1754107789 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd0800387b0 msgr2=0x7fd08003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:19.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:19.944+0000 7fd09bf36700 1 --2- 172.21.15.49:0/1754107789 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd0800387b0 0x7fd08003ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd088006fd0 tx=0x7fd088006e40 comp rx=0 tx=0).stop 2023-07-20T17:43:19.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:19.944+0000 7fd09bf36700 1 -- 172.21.15.49:0/1754107789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd094104120 msgr2=0x7fd0940fb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:19.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:19.944+0000 7fd09bf36700 1 --2- 172.21.15.49:0/1754107789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd094104120 0x7fd0940fb7b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd090004f40 tx=0x7fd090005020 comp rx=0 tx=0).stop 2023-07-20T17:43:19.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:19.944+0000 7fd09bf36700 1 -- 172.21.15.49:0/1754107789 shutdown_connections 2023-07-20T17:43:19.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:19.944+0000 7fd09bf36700 1 --2- 172.21.15.49:0/1754107789 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd0800387b0 0x7fd08003ac70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:19.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:19.944+0000 7fd09bf36700 1 --2- 172.21.15.49:0/1754107789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd094104120 0x7fd0940fb7b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:19.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:19.944+0000 7fd09bf36700 1 -- 172.21.15.49:0/1754107789 >> 172.21.15.49:0/1754107789 conn(0x7fd0940757d0 msgr2=0x7fd094102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:19.946 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:19.944+0000 7fd09bf36700 1 -- 172.21.15.49:0/1754107789 shutdown_connections 2023-07-20T17:43:19.946 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:19.944+0000 7fd09bf36700 1 -- 172.21.15.49:0/1754107789 wait complete. 2023-07-20T17:43:20.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T17:43:20.844 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:43:21.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:20 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:21.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:20 smithi049 ceph-mon[96530]: Added host smithi097 2023-07-20T17:43:21.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:20 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:21.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:20 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:21.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:20 smithi049 ceph-mon[96530]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:22.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.032+0000 7f50e1b3c700 1 -- 172.21.15.49:0/4143964013 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50dc0fe9a0 msgr2=0x7f50dc0fed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:22.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.032+0000 7f50e1b3c700 1 --2- 172.21.15.49:0/4143964013 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50dc0fe9a0 0x7f50dc0fed80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009910 tx=0x7f50cc009c20 comp rx=0 tx=0).stop 2023-07-20T17:43:22.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.032+0000 7f50e1b3c700 1 -- 172.21.15.49:0/4143964013 shutdown_connections 2023-07-20T17:43:22.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.032+0000 7f50e1b3c700 1 --2- 172.21.15.49:0/4143964013 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50dc0fe9a0 0x7f50dc0fed80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:22.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.032+0000 7f50e1b3c700 1 -- 172.21.15.49:0/4143964013 >> 172.21.15.49:0/4143964013 conn(0x7f50dc0757d0 msgr2=0x7f50dc075be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:22.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.032+0000 7f50e1b3c700 1 -- 172.21.15.49:0/4143964013 shutdown_connections 2023-07-20T17:43:22.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.032+0000 7f50e1b3c700 1 -- 172.21.15.49:0/4143964013 wait complete. 2023-07-20T17:43:22.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.033+0000 7f50e1b3c700 1 Processor -- start 2023-07-20T17:43:22.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.033+0000 7f50e1b3c700 1 -- start start 2023-07-20T17:43:22.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.033+0000 7f50e1b3c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50dc0fe9a0 0x7f50dc195710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:22.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.033+0000 7f50e1b3c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc195c50 con 0x7f50dc0fe9a0 2023-07-20T17:43:22.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.033+0000 7f50db7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50dc0fe9a0 0x7f50dc195710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:22.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.033+0000 7f50db7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50dc0fe9a0 0x7f50dc195710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35548/0 (socket says 172.21.15.49:35548) 2023-07-20T17:43:22.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.033+0000 7f50db7fe700 1 -- 172.21.15.49:0/301128748 learned_addr learned my addr 172.21.15.49:0/301128748 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:22.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.033+0000 7f50db7fe700 1 -- 172.21.15.49:0/301128748 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc0095f0 con 0x7f50dc0fe9a0 2023-07-20T17:43:22.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.034+0000 7f50db7fe700 1 --2- 172.21.15.49:0/301128748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50dc0fe9a0 0x7f50dc195710 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f50cc006010 tx=0x7f50cc004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:22.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.034+0000 7f50d8ff9700 1 -- 172.21.15.49:0/301128748 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50cc01d070 con 0x7f50dc0fe9a0 2023-07-20T17:43:22.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.034+0000 7f50d8ff9700 1 -- 172.21.15.49:0/301128748 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50cc00bc10 con 0x7f50dc0fe9a0 2023-07-20T17:43:22.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.034+0000 7f50d8ff9700 1 -- 172.21.15.49:0/301128748 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50cc003910 con 0x7f50dc0fe9a0 2023-07-20T17:43:22.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.034+0000 7f50e1b3c700 1 -- 172.21.15.49:0/301128748 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50dc195e50 con 0x7f50dc0fe9a0 2023-07-20T17:43:22.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.034+0000 7f50e1b3c700 1 -- 172.21.15.49:0/301128748 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50dc196270 con 0x7f50dc0fe9a0 2023-07-20T17:43:22.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.035+0000 7f50e1b3c700 1 -- 172.21.15.49:0/301128748 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50dc061da0 con 0x7f50dc0fe9a0 2023-07-20T17:43:22.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.035+0000 7f50d8ff9700 1 -- 172.21.15.49:0/301128748 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f50cc00bd80 con 0x7f50dc0fe9a0 2023-07-20T17:43:22.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.035+0000 7f50d8ff9700 1 --2- 172.21.15.49:0/301128748 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f50c4038800 0x7f50c403acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:22.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.035+0000 7f50d8ff9700 1 -- 172.21.15.49:0/301128748 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f50cc04c4a0 con 0x7f50dc0fe9a0 2023-07-20T17:43:22.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.035+0000 7f50daffd700 1 --2- 172.21.15.49:0/301128748 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f50c4038800 0x7f50c403acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:22.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.036+0000 7f50daffd700 1 --2- 172.21.15.49:0/301128748 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f50c4038800 0x7f50c403acc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f50d0006fd0 tx=0x7f50d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:22.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.040+0000 7f50d8ff9700 1 -- 172.21.15.49:0/301128748 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f50cc029360 con 0x7f50dc0fe9a0 2023-07-20T17:43:22.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.244+0000 7f50e1b3c700 1 -- 172.21.15.49:0/301128748 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50dc0fd940 con 0x7f50c4038800 2023-07-20T17:43:22.246 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.245+0000 7f50d8ff9700 1 -- 172.21.15.49:0/301128748 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+238 (secure 0 0 0) 0x7f50dc0fd940 con 0x7f50c4038800 2023-07-20T17:43:22.246 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:43:22.246 INFO:teuthology.orchestra.run.smithi049.stdout:[{"addr": "172.21.15.49", "hostname": "smithi049", "labels": [], "status": ""}, {"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}, {"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}] 2023-07-20T17:43:22.248 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.247+0000 7f50e1b3c700 1 -- 172.21.15.49:0/301128748 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f50c4038800 msgr2=0x7f50c403acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:22.248 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.247+0000 7f50e1b3c700 1 --2- 172.21.15.49:0/301128748 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f50c4038800 0x7f50c403acc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f50d0006fd0 tx=0x7f50d0006e40 comp rx=0 tx=0).stop 2023-07-20T17:43:22.248 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.247+0000 7f50e1b3c700 1 -- 172.21.15.49:0/301128748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50dc0fe9a0 msgr2=0x7f50dc195710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:22.248 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.247+0000 7f50e1b3c700 1 --2- 172.21.15.49:0/301128748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50dc0fe9a0 0x7f50dc195710 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f50cc006010 tx=0x7f50cc004ff0 comp rx=0 tx=0).stop 2023-07-20T17:43:22.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.247+0000 7f50e1b3c700 1 -- 172.21.15.49:0/301128748 shutdown_connections 2023-07-20T17:43:22.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.247+0000 7f50e1b3c700 1 --2- 172.21.15.49:0/301128748 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f50c4038800 0x7f50c403acc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:22.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.247+0000 7f50e1b3c700 1 --2- 172.21.15.49:0/301128748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50dc0fe9a0 0x7f50dc195710 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:22.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.247+0000 7f50e1b3c700 1 -- 172.21.15.49:0/301128748 >> 172.21.15.49:0/301128748 conn(0x7f50dc0757d0 msgr2=0x7f50dc104700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:22.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.248+0000 7f50e1b3c700 1 -- 172.21.15.49:0/301128748 shutdown_connections 2023-07-20T17:43:22.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:22.248+0000 7f50e1b3c700 1 -- 172.21.15.49:0/301128748 wait complete. 2023-07-20T17:43:22.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:22 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:22.604 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-20T17:43:22.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph osd crush tunables default 2023-07-20T17:43:23.163 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:43:23.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:23 smithi049 ceph-mon[96530]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T17:43:23.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:23.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:23 smithi049 ceph-mon[96530]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:24.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.353+0000 7f68bef4d700 1 -- 172.21.15.49:0/333203219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68b80ff340 msgr2=0x7f68b80ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:24.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.353+0000 7f68bef4d700 1 --2- 172.21.15.49:0/333203219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68b80ff340 0x7f68b80ff720 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009910 tx=0x7f68ac009c20 comp rx=0 tx=0).stop 2023-07-20T17:43:24.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.353+0000 7f68bef4d700 1 -- 172.21.15.49:0/333203219 shutdown_connections 2023-07-20T17:43:24.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.353+0000 7f68bef4d700 1 --2- 172.21.15.49:0/333203219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68b80ff340 0x7f68b80ff720 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:24.355 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.353+0000 7f68bef4d700 1 -- 172.21.15.49:0/333203219 >> 172.21.15.49:0/333203219 conn(0x7f68b80fabf0 msgr2=0x7f68b80fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:24.355 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.353+0000 7f68bef4d700 1 -- 172.21.15.49:0/333203219 shutdown_connections 2023-07-20T17:43:24.355 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.353+0000 7f68bef4d700 1 -- 172.21.15.49:0/333203219 wait complete. 2023-07-20T17:43:24.356 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.354+0000 7f68bef4d700 1 Processor -- start 2023-07-20T17:43:24.357 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.354+0000 7f68bef4d700 1 -- start start 2023-07-20T17:43:24.357 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.354+0000 7f68bef4d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68b80ff340 0x7f68b8196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:24.357 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.354+0000 7f68bef4d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8196a80 con 0x7f68b80ff340 2023-07-20T17:43:24.357 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.354+0000 7f68bcce9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68b80ff340 0x7f68b8196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:24.357 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.354+0000 7f68bcce9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68b80ff340 0x7f68b8196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35554/0 (socket says 172.21.15.49:35554) 2023-07-20T17:43:24.357 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.354+0000 7f68bcce9700 1 -- 172.21.15.49:0/199423917 learned_addr learned my addr 172.21.15.49:0/199423917 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:24.357 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.354+0000 7f68bcce9700 1 -- 172.21.15.49:0/199423917 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac0095f0 con 0x7f68b80ff340 2023-07-20T17:43:24.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.355+0000 7f68bcce9700 1 --2- 172.21.15.49:0/199423917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68b80ff340 0x7f68b8196540 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f68ac006010 tx=0x7f68ac004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:24.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.355+0000 7f68b5ffb700 1 -- 172.21.15.49:0/199423917 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68ac01d070 con 0x7f68b80ff340 2023-07-20T17:43:24.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.355+0000 7f68b5ffb700 1 -- 172.21.15.49:0/199423917 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68ac00bc10 con 0x7f68b80ff340 2023-07-20T17:43:24.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.355+0000 7f68bef4d700 1 -- 172.21.15.49:0/199423917 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b8196c80 con 0x7f68b80ff340 2023-07-20T17:43:24.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.355+0000 7f68b5ffb700 1 -- 172.21.15.49:0/199423917 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68ac003910 con 0x7f68b80ff340 2023-07-20T17:43:24.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.355+0000 7f68bef4d700 1 -- 172.21.15.49:0/199423917 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b81970a0 con 0x7f68b80ff340 2023-07-20T17:43:24.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.356+0000 7f68b5ffb700 1 -- 172.21.15.49:0/199423917 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f68ac003b80 con 0x7f68b80ff340 2023-07-20T17:43:24.359 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.356+0000 7f68bef4d700 1 -- 172.21.15.49:0/199423917 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b8107650 con 0x7f68b80ff340 2023-07-20T17:43:24.359 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.356+0000 7f68b5ffb700 1 --2- 172.21.15.49:0/199423917 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68a8038800 0x7f68a803acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:24.359 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.356+0000 7f68b5ffb700 1 -- 172.21.15.49:0/199423917 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f68ac04c460 con 0x7f68b80ff340 2023-07-20T17:43:24.359 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.356+0000 7f68b7fff700 1 --2- 172.21.15.49:0/199423917 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68a8038800 0x7f68a803acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:24.359 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.357+0000 7f68b7fff700 1 --2- 172.21.15.49:0/199423917 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68a8038800 0x7f68a803acc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f68a4006fd0 tx=0x7f68a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:24.361 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.361+0000 7f68b5ffb700 1 -- 172.21.15.49:0/199423917 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f68ac029950 con 0x7f68b80ff340 2023-07-20T17:43:24.562 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.561+0000 7f68bef4d700 1 -- 172.21.15.49:0/199423917 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f68b80687f0 con 0x7f68b80ff340 2023-07-20T17:43:24.614 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.614+0000 7f68b5ffb700 1 -- 172.21.15.49:0/199423917 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f68ac026020 con 0x7f68b80ff340 2023-07-20T17:43:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.616+0000 7f68bef4d700 1 -- 172.21.15.49:0/199423917 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68a8038800 msgr2=0x7f68a803acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.616+0000 7f68bef4d700 1 --2- 172.21.15.49:0/199423917 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68a8038800 0x7f68a803acc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f68a4006fd0 tx=0x7f68a4006e40 comp rx=0 tx=0).stop 2023-07-20T17:43:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.616+0000 7f68bef4d700 1 -- 172.21.15.49:0/199423917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68b80ff340 msgr2=0x7f68b8196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.616+0000 7f68bef4d700 1 --2- 172.21.15.49:0/199423917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68b80ff340 0x7f68b8196540 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f68ac006010 tx=0x7f68ac004ff0 comp rx=0 tx=0).stop 2023-07-20T17:43:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.616+0000 7f68bef4d700 1 -- 172.21.15.49:0/199423917 shutdown_connections 2023-07-20T17:43:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.616+0000 7f68bef4d700 1 --2- 172.21.15.49:0/199423917 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68a8038800 0x7f68a803acc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.616+0000 7f68bef4d700 1 --2- 172.21.15.49:0/199423917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68b80ff340 0x7f68b8196540 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.616+0000 7f68bef4d700 1 -- 172.21.15.49:0/199423917 >> 172.21.15.49:0/199423917 conn(0x7f68b80fabf0 msgr2=0x7f68b8103610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.617+0000 7f68bef4d700 1 -- 172.21.15.49:0/199423917 shutdown_connections 2023-07-20T17:43:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:24.617+0000 7f68bef4d700 1 -- 172.21.15.49:0/199423917 wait complete. 2023-07-20T17:43:24.619 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2023-07-20T17:43:24.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:24 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/199423917' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T17:43:25.018 INFO:tasks.cephadm:Adding mon.a on smithi049 2023-07-20T17:43:25.018 INFO:tasks.cephadm:Adding mon.b on smithi079 2023-07-20T17:43:25.019 INFO:tasks.cephadm:Adding mon.c on smithi097 2023-07-20T17:43:25.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch apply mon '3;smithi049:172.21.15.49=a;smithi079:172.21.15.79=b;smithi097:172.21.15.97=c' 2023-07-20T17:43:25.609 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:25.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:25 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/199423917' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T17:43:25.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:25 smithi049 ceph-mon[96530]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T17:43:25.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:25 smithi049 ceph-mon[96530]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:25.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:25.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:25.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:25.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:25.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:43:25.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:25.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:26.854 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.853+0000 7f3583574700 1 -- 172.21.15.97:0/2198618165 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357c105180 msgr2=0x7f357c105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:26.855 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.853+0000 7f3583574700 1 --2- 172.21.15.97:0/2198618165 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357c105180 0x7f357c105560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f356c009910 tx=0x7f356c009c20 comp rx=0 tx=0).stop 2023-07-20T17:43:26.855 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.853+0000 7f3583574700 1 -- 172.21.15.97:0/2198618165 shutdown_connections 2023-07-20T17:43:26.855 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.853+0000 7f3583574700 1 --2- 172.21.15.97:0/2198618165 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357c105180 0x7f357c105560 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:26.855 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.853+0000 7f3583574700 1 -- 172.21.15.97:0/2198618165 >> 172.21.15.97:0/2198618165 conn(0x7f357c074c70 msgr2=0x7f357c075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:26.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.854+0000 7f3583574700 1 -- 172.21.15.97:0/2198618165 shutdown_connections 2023-07-20T17:43:26.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.854+0000 7f3583574700 1 -- 172.21.15.97:0/2198618165 wait complete. 2023-07-20T17:43:26.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.854+0000 7f3583574700 1 Processor -- start 2023-07-20T17:43:26.858 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.854+0000 7f3583574700 1 -- start start 2023-07-20T17:43:26.858 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.855+0000 7f3583574700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357c105180 0x7f357c1964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:26.858 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.855+0000 7f3583574700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357c196a20 con 0x7f357c105180 2023-07-20T17:43:26.858 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.855+0000 7f3581310700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357c105180 0x7f357c1964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:26.858 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.855+0000 7f3581310700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357c105180 0x7f357c1964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.97:32998/0 (socket says 172.21.15.97:32998) 2023-07-20T17:43:26.858 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.855+0000 7f3581310700 1 -- 172.21.15.97:0/3515471930 learned_addr learned my addr 172.21.15.97:0/3515471930 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-07-20T17:43:26.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.855+0000 7f3581310700 1 -- 172.21.15.97:0/3515471930 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0095f0 con 0x7f357c105180 2023-07-20T17:43:26.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.855+0000 7f3581310700 1 --2- 172.21.15.97:0/3515471930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357c105180 0x7f357c1964e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f356c006010 tx=0x7f356c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:26.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.855+0000 7f35727fc700 1 -- 172.21.15.97:0/3515471930 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f356c01d070 con 0x7f357c105180 2023-07-20T17:43:26.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.855+0000 7f35727fc700 1 -- 172.21.15.97:0/3515471930 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f356c00bbb0 con 0x7f357c105180 2023-07-20T17:43:26.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.855+0000 7f3583574700 1 -- 172.21.15.97:0/3515471930 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f357c196c80 con 0x7f357c105180 2023-07-20T17:43:26.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.855+0000 7f35727fc700 1 -- 172.21.15.97:0/3515471930 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f356c003930 con 0x7f357c105180 2023-07-20T17:43:26.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.856+0000 7f3583574700 1 -- 172.21.15.97:0/3515471930 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f357c197100 con 0x7f357c105180 2023-07-20T17:43:26.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.856+0000 7f35727fc700 1 -- 172.21.15.97:0/3515471930 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f356c00bd20 con 0x7f357c105180 2023-07-20T17:43:26.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.856+0000 7f3583574700 1 -- 172.21.15.97:0/3515471930 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f357c197340 con 0x7f357c105180 2023-07-20T17:43:26.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.857+0000 7f35727fc700 1 --2- 172.21.15.97:0/3515471930 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f35680387b0 0x7f356803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:26.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.857+0000 7f35727fc700 1 -- 172.21.15.97:0/3515471930 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f356c04c570 con 0x7f357c105180 2023-07-20T17:43:26.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.857+0000 7f3580b0f700 1 --2- 172.21.15.97:0/3515471930 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f35680387b0 0x7f356803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:26.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.857+0000 7f3580b0f700 1 --2- 172.21.15.97:0/3515471930 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f35680387b0 0x7f356803ac70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3578006fd0 tx=0x7f3578006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:26.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:26.862+0000 7f35727fc700 1 -- 172.21.15.97:0/3515471930 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f356c029360 con 0x7f357c105180 2023-07-20T17:43:26.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:26 smithi049 ceph-mon[96530]: Updating smithi097:/etc/ceph/ceph.conf 2023-07-20T17:43:26.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:26 smithi049 ceph-mon[96530]: Updating smithi097:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:26.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:26 smithi049 ceph-mon[96530]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:26.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:26 smithi049 ceph-mon[96530]: Updating smithi097:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T17:43:26.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:26.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:26.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:27.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.058+0000 7f3583574700 1 -- 172.21.15.97:0/3515471930 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi079:172.21.15.79=b;smithi097:172.21.15.97=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f357c068160 con 0x7f35680387b0 2023-07-20T17:43:27.066 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.066+0000 7f35727fc700 1 -- 172.21.15.97:0/3515471930 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f357c068160 con 0x7f35680387b0 2023-07-20T17:43:27.066 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled mon update... 2023-07-20T17:43:27.069 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.068+0000 7f3583574700 1 -- 172.21.15.97:0/3515471930 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f35680387b0 msgr2=0x7f356803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:27.069 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.068+0000 7f3583574700 1 --2- 172.21.15.97:0/3515471930 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f35680387b0 0x7f356803ac70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3578006fd0 tx=0x7f3578006e40 comp rx=0 tx=0).stop 2023-07-20T17:43:27.069 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.068+0000 7f3583574700 1 -- 172.21.15.97:0/3515471930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357c105180 msgr2=0x7f357c1964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:27.069 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.068+0000 7f3583574700 1 --2- 172.21.15.97:0/3515471930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357c105180 0x7f357c1964e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f356c006010 tx=0x7f356c004ff0 comp rx=0 tx=0).stop 2023-07-20T17:43:27.069 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.068+0000 7f3583574700 1 -- 172.21.15.97:0/3515471930 shutdown_connections 2023-07-20T17:43:27.070 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.068+0000 7f3583574700 1 --2- 172.21.15.97:0/3515471930 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f35680387b0 0x7f356803ac70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:27.070 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.068+0000 7f3583574700 1 --2- 172.21.15.97:0/3515471930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357c105180 0x7f357c1964e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:27.070 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.068+0000 7f3583574700 1 -- 172.21.15.97:0/3515471930 >> 172.21.15.97:0/3515471930 conn(0x7f357c074c70 msgr2=0x7f357c0fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:27.070 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.068+0000 7f3583574700 1 -- 172.21.15.97:0/3515471930 shutdown_connections 2023-07-20T17:43:27.070 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:27.068+0000 7f3583574700 1 -- 172.21.15.97:0/3515471930 wait complete. 2023-07-20T17:43:27.489 DEBUG:teuthology.orchestra.run.smithi079:mon.b> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.b.service 2023-07-20T17:43:27.493 DEBUG:teuthology.orchestra.run.smithi097:mon.c> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.c.service 2023-07-20T17:43:27.498 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T17:43:27.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T17:43:27.525 INFO:journalctl@ceph.mon.c.smithi097.stdout:-- Logs begin at Thu 2023-07-20 17:15:39 UTC. -- 2023-07-20T17:43:27.527 INFO:journalctl@ceph.mon.b.smithi079.stdout:-- Logs begin at Thu 2023-07-20 17:15:42 UTC. -- 2023-07-20T17:43:27.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:27 smithi049 ceph-mon[96530]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:27.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:27 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:27.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:27 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:27.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:27 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:27.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:27 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:27.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:27 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:27.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:27 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:27.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:27 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:28.211 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:28.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:28 smithi049 ceph-mon[96530]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi079:172.21.15.79=b;smithi097:172.21.15.97=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:28.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:28 smithi049 ceph-mon[96530]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi079:172.21.15.79=b;smithi097:172.21.15.97=c;count:3 2023-07-20T17:43:28.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:28 smithi049 ceph-mon[96530]: Deploying daemon mon.c on smithi097 2023-07-20T17:43:29.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:29 smithi049 ceph-mon[96530]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:30.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.356+0000 7fbdaf662700 1 -- 172.21.15.97:0/782882810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda80fe960 msgr2=0x7fbda80fed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:30.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.356+0000 7fbdaf662700 1 --2- 172.21.15.97:0/782882810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda80fe960 0x7fbda80fed40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009910 tx=0x7fbda4009c20 comp rx=0 tx=0).stop 2023-07-20T17:43:30.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.357+0000 7fbdaf662700 1 -- 172.21.15.97:0/782882810 shutdown_connections 2023-07-20T17:43:30.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.357+0000 7fbdaf662700 1 --2- 172.21.15.97:0/782882810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda80fe960 0x7fbda80fed40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:30.360 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.357+0000 7fbdaf662700 1 -- 172.21.15.97:0/782882810 >> 172.21.15.97:0/782882810 conn(0x7fbda80757d0 msgr2=0x7fbda8075be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:30.360 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.357+0000 7fbdaf662700 1 -- 172.21.15.97:0/782882810 shutdown_connections 2023-07-20T17:43:30.360 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.357+0000 7fbdaf662700 1 -- 172.21.15.97:0/782882810 wait complete. 2023-07-20T17:43:30.360 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.357+0000 7fbdaf662700 1 Processor -- start 2023-07-20T17:43:30.360 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.357+0000 7fbdaf662700 1 -- start start 2023-07-20T17:43:30.360 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.358+0000 7fbdaf662700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda80fe960 0x7fbda81934a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:30.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.358+0000 7fbdaf662700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda81939e0 con 0x7fbda80fe960 2023-07-20T17:43:30.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.358+0000 7fbdad3fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda80fe960 0x7fbda81934a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:30.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.358+0000 7fbdad3fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda80fe960 0x7fbda81934a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.97:33004/0 (socket says 172.21.15.97:33004) 2023-07-20T17:43:30.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.358+0000 7fbdad3fe700 1 -- 172.21.15.97:0/767448483 learned_addr learned my addr 172.21.15.97:0/767448483 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-07-20T17:43:30.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.358+0000 7fbdad3fe700 1 -- 172.21.15.97:0/767448483 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda40095f0 con 0x7fbda80fe960 2023-07-20T17:43:30.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.358+0000 7fbdad3fe700 1 --2- 172.21.15.97:0/767448483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda80fe960 0x7fbda81934a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbda4004f70 tx=0x7fbda4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:30.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.358+0000 7fbd9e7fc700 1 -- 172.21.15.97:0/767448483 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbda401d070 con 0x7fbda80fe960 2023-07-20T17:43:30.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.359+0000 7fbdaf662700 1 -- 172.21.15.97:0/767448483 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda8193be0 con 0x7fbda80fe960 2023-07-20T17:43:30.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.359+0000 7fbd9e7fc700 1 -- 172.21.15.97:0/767448483 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbda400bc10 con 0x7fbda80fe960 2023-07-20T17:43:30.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.359+0000 7fbd9e7fc700 1 -- 172.21.15.97:0/767448483 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbda4003970 con 0x7fbda80fe960 2023-07-20T17:43:30.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.359+0000 7fbdaf662700 1 -- 172.21.15.97:0/767448483 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda8194000 con 0x7fbda80fe960 2023-07-20T17:43:30.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.359+0000 7fbdaf662700 1 -- 172.21.15.97:0/767448483 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda8061da0 con 0x7fbda80fe960 2023-07-20T17:43:30.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.360+0000 7fbd9e7fc700 1 -- 172.21.15.97:0/767448483 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fbda4004390 con 0x7fbda80fe960 2023-07-20T17:43:30.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.360+0000 7fbd9e7fc700 1 --2- 172.21.15.97:0/767448483 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbd9403cc50 0x7fbd9403f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:30.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.360+0000 7fbd9e7fc700 1 -- 172.21.15.97:0/767448483 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbda404cb50 con 0x7fbda80fe960 2023-07-20T17:43:30.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.361+0000 7fbdacbfd700 1 --2- 172.21.15.97:0/767448483 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbd9403cc50 0x7fbd9403f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:30.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.361+0000 7fbdacbfd700 1 --2- 172.21.15.97:0/767448483 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbd9403cc50 0x7fbd9403f110 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbd98006fd0 tx=0x7fbd98006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:30.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.365+0000 7fbd9e7fc700 1 -- 172.21.15.97:0/767448483 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbda400e3e0 con 0x7fbda80fe960 2023-07-20T17:43:30.644 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.643+0000 7fbdaf662700 1 -- 172.21.15.97:0/767448483 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbda818cf20 con 0x7fbda80fe960 2023-07-20T17:43:30.644 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.643+0000 7fbd9e7fc700 1 -- 172.21.15.97:0/767448483 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7fbda4026030 con 0x7fbda80fe960 2023-07-20T17:43:30.645 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:43:30.645 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"8c050cea-2724-11ee-9b34-001a4aab830c","modified":"2023-07-20T17:41:55.142063Z","created":"2023-07-20T17:41:55.142063Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-20T17:43:30.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.646+0000 7fbdaf662700 1 -- 172.21.15.97:0/767448483 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbd9403cc50 msgr2=0x7fbd9403f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:30.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.646+0000 7fbdaf662700 1 --2- 172.21.15.97:0/767448483 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbd9403cc50 0x7fbd9403f110 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbd98006fd0 tx=0x7fbd98006e40 comp rx=0 tx=0).stop 2023-07-20T17:43:30.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.646+0000 7fbdaf662700 1 -- 172.21.15.97:0/767448483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda80fe960 msgr2=0x7fbda81934a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:30.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.646+0000 7fbdaf662700 1 --2- 172.21.15.97:0/767448483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda80fe960 0x7fbda81934a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbda4004f70 tx=0x7fbda4005050 comp rx=0 tx=0).stop 2023-07-20T17:43:30.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.646+0000 7fbdaf662700 1 -- 172.21.15.97:0/767448483 shutdown_connections 2023-07-20T17:43:30.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.646+0000 7fbdaf662700 1 --2- 172.21.15.97:0/767448483 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fbd9403cc50 0x7fbd9403f110 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:30.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.646+0000 7fbdaf662700 1 --2- 172.21.15.97:0/767448483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda80fe960 0x7fbda81934a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:30.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.646+0000 7fbdaf662700 1 -- 172.21.15.97:0/767448483 >> 172.21.15.97:0/767448483 conn(0x7fbda80757d0 msgr2=0x7fbda81046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:30.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.646+0000 7fbdaf662700 1 -- 172.21.15.97:0/767448483 shutdown_connections 2023-07-20T17:43:30.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:30.646+0000 7fbdaf662700 1 -- 172.21.15.97:0/767448483 wait complete. 2023-07-20T17:43:30.648 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2023-07-20T17:43:30.802 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T17:43:30.802 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T17:43:30.803 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T17:43:30.803 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:30.803 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T17:43:30.803 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:30.803 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:30.803 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:30.803 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:30.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:30.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:30.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:43:30.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:30.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:30.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Updating smithi079:/etc/ceph/ceph.conf 2023-07-20T17:43:30.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi097", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:30.805 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Updating smithi079:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:30.805 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:30.805 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Deploying cephadm binary to smithi097 2023-07-20T17:43:30.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:30 smithi049 ceph-mon[96530]: from='client.? 172.21.15.97:0/767448483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T17:43:31.244 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Updating smithi079:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T17:43:31.244 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.245 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.245 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.245 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:31.246 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.246 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Added host smithi097 2023-07-20T17:43:31.246 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:31.246 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.247 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:31.247 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.247 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T17:43:31.247 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.248 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:31.248 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/199423917' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T17:43:31.248 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/199423917' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T17:43:31.249 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T17:43:31.249 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:31.249 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.249 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.250 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.250 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.250 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:43:31.251 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:31.251 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:31.251 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Updating smithi097:/etc/ceph/ceph.conf 2023-07-20T17:43:31.252 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Updating smithi097:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:31.252 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:31.252 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Updating smithi097:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T17:43:31.252 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.253 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.253 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.253 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:31.253 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.254 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:31.254 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:31.254 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:31.254 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:31.254 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:31.255 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:31.255 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi079:172.21.15.79=b;smithi097:172.21.15.97=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:31.255 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi079:172.21.15.79=b;smithi097:172.21.15.97=c;count:3 2023-07-20T17:43:31.256 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: Deploying daemon mon.c on smithi097 2023-07-20T17:43:31.256 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:31.256 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: from='client.? 172.21.15.97:0/767448483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T17:43:31.256 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T17:43:31.257 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-20T17:43:31.257 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta expand map: {default=false} 2023-07-20T17:43:31.257 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta from 'false' to 'false' 2023-07-20T17:43:31.257 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta expanded map: {default=false} 2023-07-20T17:43:31.257 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta expand map: {default=info} 2023-07-20T17:43:31.257 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta from 'info' to 'info' 2023-07-20T17:43:31.258 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta expanded map: {default=info} 2023-07-20T17:43:31.258 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta expand map: {default=daemon} 2023-07-20T17:43:31.258 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T17:43:31.258 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T17:43:31.258 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta expand map: {default=debug} 2023-07-20T17:43:31.258 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T17:43:31.258 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 ceph-mon[98398]: expand_channel_meta expanded map: {default=debug} 2023-07-20T17:43:31.259 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:30 smithi097 systemd[1]: Started Ceph mon.c for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:43:32.325 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T17:43:32.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T17:43:32.888 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:43:36.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:43:36.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: mon.a calling monitor election 2023-07-20T17:43:36.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:36.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: mon.c calling monitor election 2023-07-20T17:43:36.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:36.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T17:43:36.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: monmap e2: 2 mons at {a=[v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0],c=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0]} removed_ranks: {} 2023-07-20T17:43:36.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: fsmap 2023-07-20T17:43:36.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T17:43:36.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: mgrmap e13: x(active, since 43s) 2023-07-20T17:43:36.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: overall HEALTH_OK 2023-07-20T17:43:36.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:36.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:36.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:36.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:36.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:36.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:43:36.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: mon.a calling monitor election 2023-07-20T17:43:36.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:36.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: mon.c calling monitor election 2023-07-20T17:43:36.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:36.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:36.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T17:43:36.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: monmap e2: 2 mons at {a=[v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0],c=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0]} removed_ranks: {} 2023-07-20T17:43:36.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: fsmap 2023-07-20T17:43:36.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T17:43:36.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: mgrmap e13: x(active, since 43s) 2023-07-20T17:43:36.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: overall HEALTH_OK 2023-07-20T17:43:36.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:36.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:36.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:36.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:36.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:37.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.093+0000 7f9dff256700 1 -- 172.21.15.97:0/2232916809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9de80035c0 msgr2=0x7f9de8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:37.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.093+0000 7f9dff256700 1 --2- 172.21.15.97:0/2232916809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9de80035c0 0x7f9de8005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9df8075870 tx=0x7f9dec00feb0 comp rx=0 tx=0).stop 2023-07-20T17:43:37.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.093+0000 7f9dff256700 1 -- 172.21.15.97:0/2232916809 shutdown_connections 2023-07-20T17:43:37.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.093+0000 7f9dff256700 1 --2- 172.21.15.97:0/2232916809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9de80035c0 0x7f9de8005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:37.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.093+0000 7f9dff256700 1 -- 172.21.15.97:0/2232916809 >> 172.21.15.97:0/2232916809 conn(0x7f9df80744c0 msgr2=0x7f9df80748d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:37.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.093+0000 7f9dff256700 1 -- 172.21.15.97:0/2232916809 shutdown_connections 2023-07-20T17:43:37.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.093+0000 7f9dff256700 1 -- 172.21.15.97:0/2232916809 wait complete. 2023-07-20T17:43:37.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9dff256700 1 Processor -- start 2023-07-20T17:43:37.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9dff256700 1 -- start start 2023-07-20T17:43:37.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9dff256700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9de80035c0 0x7f9df81959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:37.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9dff256700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9df8195ee0 0x7f9df819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:37.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9dff256700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8196470 con 0x7f9de80035c0 2023-07-20T17:43:37.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9dff256700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81965b0 con 0x7f9df8195ee0 2023-07-20T17:43:37.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9df7fff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9df8195ee0 0x7f9df819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:37.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9df7fff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9df8195ee0 0x7f9df819a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60600/0 (socket says 172.21.15.97:60600) 2023-07-20T17:43:37.098 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9df7fff700 1 -- 172.21.15.97:0/1213479966 learned_addr learned my addr 172.21.15.97:0/1213479966 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-07-20T17:43:37.098 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9dfcff2700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9de80035c0 0x7f9df81959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:37.098 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9df7fff700 1 -- 172.21.15.97:0/1213479966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9df8195ee0 msgr2=0x7f9df819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-20T17:43:37.098 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9df7fff700 1 -- 172.21.15.97:0/1213479966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9df8195ee0 msgr2=0x7f9df819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T17:43:37.098 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9df7fff700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9df8195ee0 0x7f9df819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T17:43:37.098 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.095+0000 7f9df7fff700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9df8195ee0 0x7f9df819a380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T17:43:37.098 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.096+0000 7f9dfcff2700 1 -- 172.21.15.97:0/1213479966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9df8195ee0 msgr2=0x7f9df819a380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T17:43:37.099 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.096+0000 7f9dfcff2700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9df8195ee0 0x7f9df819a380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:37.099 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.096+0000 7f9dfcff2700 1 -- 172.21.15.97:0/1213479966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec00f900 con 0x7f9de80035c0 2023-07-20T17:43:37.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.096+0000 7f9dfcff2700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9de80035c0 0x7f9df81959a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9dec011510 tx=0x7f9dec004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:37.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.096+0000 7f9df5ffb700 1 -- 172.21.15.97:0/1213479966 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f9dec013b00 con 0x7f9de80035c0 2023-07-20T17:43:37.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.096+0000 7f9df5ffb700 1 -- 172.21.15.97:0/1213479966 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9dec0044f0 con 0x7f9de80035c0 2023-07-20T17:43:37.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.096+0000 7f9dff256700 1 -- 172.21.15.97:0/1213479966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df819a8c0 con 0x7f9de80035c0 2023-07-20T17:43:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.096+0000 7f9df5ffb700 1 -- 172.21.15.97:0/1213479966 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f9dec003c20 con 0x7f9de80035c0 2023-07-20T17:43:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.097+0000 7f9dff256700 1 -- 172.21.15.97:0/1213479966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df819ae10 con 0x7f9de80035c0 2023-07-20T17:43:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.098+0000 7f9dff256700 1 -- 172.21.15.97:0/1213479966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df8061da0 con 0x7f9de80035c0 2023-07-20T17:43:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.098+0000 7f9df5ffb700 1 -- 172.21.15.97:0/1213479966 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f9dec013cf0 con 0x7f9de80035c0 2023-07-20T17:43:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.098+0000 7f9df5ffb700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9ddc0385e0 0x7f9ddc03aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.098+0000 7f9df5ffb700 1 -- 172.21.15.97:0/1213479966 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9dec04dd40 con 0x7f9de80035c0 2023-07-20T17:43:37.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.098+0000 7f9df7fff700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9ddc0385e0 0x7f9ddc03aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:37.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.099+0000 7f9df7fff700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9ddc0385e0 0x7f9ddc03aaa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009180 tx=0x7f9de4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:37.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.103+0000 7f9df5ffb700 1 -- 172.21.15.97:0/1213479966 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9dec034350 con 0x7f9de80035c0 2023-07-20T17:43:37.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:36 smithi049 ceph-mon[96530]: Deploying daemon mon.b on smithi079 2023-07-20T17:43:37.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:36 smithi049 ceph-mon[96530]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:37.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:37.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:36 smithi097 ceph-mon[98398]: Deploying daemon mon.b on smithi079 2023-07-20T17:43:37.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:36 smithi097 ceph-mon[98398]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:37.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:37.369 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.368+0000 7f9dff256700 1 -- 172.21.15.97:0/1213479966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9df819b0e0 con 0x7f9de80035c0 2023-07-20T17:43:37.370 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.369+0000 7f9df5ffb700 1 -- 172.21.15.97:0/1213479966 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1002 (secure 0 0 0) 0x7f9dec034350 con 0x7f9de80035c0 2023-07-20T17:43:37.370 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:43:37.370 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":2,"fsid":"8c050cea-2724-11ee-9b34-001a4aab830c","modified":"2023-07-20T17:43:30.807366Z","created":"2023-07-20T17:41:55.142063Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T17:43:37.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.371+0000 7f9dff256700 1 -- 172.21.15.97:0/1213479966 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9ddc0385e0 msgr2=0x7f9ddc03aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:37.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.371+0000 7f9dff256700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9ddc0385e0 0x7f9ddc03aaa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009180 tx=0x7f9de4008040 comp rx=0 tx=0).stop 2023-07-20T17:43:37.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.371+0000 7f9dff256700 1 -- 172.21.15.97:0/1213479966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9de80035c0 msgr2=0x7f9df81959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:37.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.371+0000 7f9dff256700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9de80035c0 0x7f9df81959a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9dec011510 tx=0x7f9dec004f80 comp rx=0 tx=0).stop 2023-07-20T17:43:37.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.371+0000 7f9dff256700 1 -- 172.21.15.97:0/1213479966 shutdown_connections 2023-07-20T17:43:37.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.371+0000 7f9dff256700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9ddc0385e0 0x7f9ddc03aaa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:37.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.371+0000 7f9dff256700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9de80035c0 0x7f9df81959a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:37.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.371+0000 7f9dff256700 1 --2- 172.21.15.97:0/1213479966 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9df8195ee0 0x7f9df819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:37.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.371+0000 7f9dff256700 1 -- 172.21.15.97:0/1213479966 >> 172.21.15.97:0/1213479966 conn(0x7f9df80744c0 msgr2=0x7f9df818d750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:37.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.371+0000 7f9dff256700 1 -- 172.21.15.97:0/1213479966 shutdown_connections 2023-07-20T17:43:37.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:37.371+0000 7f9dff256700 1 -- 172.21.15.97:0/1213479966 wait complete. 2023-07-20T17:43:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 2 2023-07-20T17:43:38.238 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:43:37 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:43:37.807+0000 7f0405ab6700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-07-20T17:43:38.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:38 smithi049 ceph-mon[96530]: from='client.? 172.21.15.97:0/1213479966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T17:43:38.747 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 systemd[1]: Starting Ceph mon.b for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:43:38.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:38 smithi097 ceph-mon[98398]: from='client.? 172.21.15.97:0/1213479966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T17:43:38.837 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T17:43:38.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T17:43:39.003 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 podman[97996]: 2023-07-20T17:43:39.003 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 bash[97996]: ce88b8e1226172a3319795cc6bc8f81c31e1a60a551495762506644ca730c262 2023-07-20T17:43:39.003 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T17:43:39.003 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T17:43:39.004 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pidfile_write: ignore empty --pid-file 2023-07-20T17:43:39.004 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: load: jerasure load: lrc 2023-07-20T17:43:39.004 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: RocksDB version: 7.9.2 2023-07-20T17:43:39.004 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Git sha 0 2023-07-20T17:43:39.004 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T17:43:39.004 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: DB SUMMARY 2023-07-20T17:43:39.004 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: DB Session ID: 0IZZA78R2OW3DCKZ199U 2023-07-20T17:43:39.005 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: CURRENT file: CURRENT 2023-07-20T17:43:39.005 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T17:43:39.005 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T17:43:39.005 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-20T17:43:39.005 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-20T17:43:39.005 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.error_if_exists: 0 2023-07-20T17:43:39.005 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.create_if_missing: 0 2023-07-20T17:43:39.005 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.paranoid_checks: 1 2023-07-20T17:43:39.006 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T17:43:39.006 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T17:43:39.006 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T17:43:39.006 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.env: 0x56297be7bc60 2023-07-20T17:43:39.006 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T17:43:39.006 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.info_log: 0x56297d13d2c0 2023-07-20T17:43:39.006 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T17:43:39.006 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.statistics: (nil) 2023-07-20T17:43:39.007 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.use_fsync: 0 2023-07-20T17:43:39.007 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_log_file_size: 0 2023-07-20T17:43:39.007 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T17:43:39.007 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T17:43:39.007 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T17:43:39.007 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T17:43:39.007 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.allow_fallocate: 1 2023-07-20T17:43:39.007 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T17:43:39.008 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T17:43:39.008 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.use_direct_reads: 0 2023-07-20T17:43:39.008 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T17:43:39.008 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T17:43:39.008 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.db_log_dir: 2023-07-20T17:43:39.008 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.wal_dir: 2023-07-20T17:43:39.008 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T17:43:39.008 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T17:43:39.009 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T17:43:39.009 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T17:43:39.009 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T17:43:39.009 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T17:43:39.009 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T17:43:39.009 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T17:43:39.009 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.write_buffer_manager: 0x56297d05e3c0 2023-07-20T17:43:39.009 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T17:43:39.010 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T17:43:39.010 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T17:43:39.010 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T17:43:39.010 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T17:43:39.010 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T17:43:39.010 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T17:43:39.010 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T17:43:39.010 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.unordered_write: 0 2023-07-20T17:43:39.011 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T17:43:39.011 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T17:43:39.011 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T17:43:39.011 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T17:43:39.011 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.row_cache: None 2023-07-20T17:43:39.011 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.wal_filter: None 2023-07-20T17:43:39.011 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T17:43:39.011 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T17:43:39.012 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.two_write_queues: 0 2023-07-20T17:43:39.012 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T17:43:39.012 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.wal_compression: 0 2023-07-20T17:43:39.012 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.atomic_flush: 0 2023-07-20T17:43:39.012 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T17:43:39.012 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T17:43:39.012 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T17:43:39.012 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.log_readahead_size: 0 2023-07-20T17:43:39.013 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T17:43:39.013 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T17:43:39.013 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T17:43:39.013 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T17:43:39.013 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T17:43:39.014 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T17:43:39.015 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T17:43:39.015 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_background_jobs: 2 2023-07-20T17:43:39.015 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_background_compactions: -1 2023-07-20T17:43:39.015 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_subcompactions: 1 2023-07-20T17:43:39.015 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T17:43:39.015 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T17:43:39.015 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T17:43:39.016 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T17:43:39.016 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T17:43:39.016 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T17:43:39.016 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T17:43:39.016 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T17:43:39.016 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_open_files: -1 2023-07-20T17:43:39.016 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T17:43:39.016 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T17:43:39.017 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T17:43:39.017 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T17:43:39.017 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_background_flushes: -1 2023-07-20T17:43:39.017 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Compression algorithms supported: 2023-07-20T17:43:39.017 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T17:43:39.017 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: kZSTD supported: 0 2023-07-20T17:43:39.017 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: kXpressCompression supported: 0 2023-07-20T17:43:39.017 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T17:43:39.018 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: kZlibCompression supported: 1 2023-07-20T17:43:39.018 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: kSnappyCompression supported: 1 2023-07-20T17:43:39.018 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: kLZ4Compression supported: 1 2023-07-20T17:43:39.018 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: kBZip2Compression supported: 0 2023-07-20T17:43:39.018 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T17:43:39.018 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T17:43:39.018 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-20T17:43:39.018 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T17:43:39.019 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T17:43:39.019 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.merge_operator: 2023-07-20T17:43:39.019 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_filter: None 2023-07-20T17:43:39.019 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T17:43:39.019 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T17:43:39.019 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T17:43:39.019 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T17:43:39.019 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56297d13d3c0) 2023-07-20T17:43:39.020 INFO:journalctl@ceph.mon.b.smithi079.stdout: cache_index_and_filter_blocks: 1 2023-07-20T17:43:39.020 INFO:journalctl@ceph.mon.b.smithi079.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T17:43:39.020 INFO:journalctl@ceph.mon.b.smithi079.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T17:43:39.020 INFO:journalctl@ceph.mon.b.smithi079.stdout: pin_top_level_index_and_filter: 1 2023-07-20T17:43:39.020 INFO:journalctl@ceph.mon.b.smithi079.stdout: index_type: 0 2023-07-20T17:43:39.020 INFO:journalctl@ceph.mon.b.smithi079.stdout: data_block_index_type: 0 2023-07-20T17:43:39.020 INFO:journalctl@ceph.mon.b.smithi079.stdout: index_shortening: 1 2023-07-20T17:43:39.021 INFO:journalctl@ceph.mon.b.smithi079.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T17:43:39.021 INFO:journalctl@ceph.mon.b.smithi079.stdout: checksum: 4 2023-07-20T17:43:39.021 INFO:journalctl@ceph.mon.b.smithi079.stdout: no_block_cache: 0 2023-07-20T17:43:39.021 INFO:journalctl@ceph.mon.b.smithi079.stdout: block_cache: 0x56297d0fd350 2023-07-20T17:43:39.021 INFO:journalctl@ceph.mon.b.smithi079.stdout: block_cache_name: BinnedLRUCache 2023-07-20T17:43:39.021 INFO:journalctl@ceph.mon.b.smithi079.stdout: block_cache_options: 2023-07-20T17:43:39.021 INFO:journalctl@ceph.mon.b.smithi079.stdout: capacity : 536870912 2023-07-20T17:43:39.021 INFO:journalctl@ceph.mon.b.smithi079.stdout: num_shard_bits : 4 2023-07-20T17:43:39.022 INFO:journalctl@ceph.mon.b.smithi079.stdout: strict_capacity_limit : 0 2023-07-20T17:43:39.022 INFO:journalctl@ceph.mon.b.smithi079.stdout: high_pri_pool_ratio: 0.000 2023-07-20T17:43:39.022 INFO:journalctl@ceph.mon.b.smithi079.stdout: block_cache_compressed: (nil) 2023-07-20T17:43:39.022 INFO:journalctl@ceph.mon.b.smithi079.stdout: persistent_cache: (nil) 2023-07-20T17:43:39.022 INFO:journalctl@ceph.mon.b.smithi079.stdout: block_size: 4096 2023-07-20T17:43:39.022 INFO:journalctl@ceph.mon.b.smithi079.stdout: block_size_deviation: 10 2023-07-20T17:43:39.022 INFO:journalctl@ceph.mon.b.smithi079.stdout: block_restart_interval: 16 2023-07-20T17:43:39.022 INFO:journalctl@ceph.mon.b.smithi079.stdout: index_block_restart_interval: 1 2023-07-20T17:43:39.022 INFO:journalctl@ceph.mon.b.smithi079.stdout: metadata_block_size: 4096 2023-07-20T17:43:39.023 INFO:journalctl@ceph.mon.b.smithi079.stdout: partition_filters: 0 2023-07-20T17:43:39.023 INFO:journalctl@ceph.mon.b.smithi079.stdout: use_delta_encoding: 1 2023-07-20T17:43:39.023 INFO:journalctl@ceph.mon.b.smithi079.stdout: filter_policy: bloomfilter 2023-07-20T17:43:39.023 INFO:journalctl@ceph.mon.b.smithi079.stdout: whole_key_filtering: 1 2023-07-20T17:43:39.023 INFO:journalctl@ceph.mon.b.smithi079.stdout: verify_compression: 0 2023-07-20T17:43:39.023 INFO:journalctl@ceph.mon.b.smithi079.stdout: read_amp_bytes_per_bit: 0 2023-07-20T17:43:39.023 INFO:journalctl@ceph.mon.b.smithi079.stdout: format_version: 5 2023-07-20T17:43:39.023 INFO:journalctl@ceph.mon.b.smithi079.stdout: enable_index_compression: 1 2023-07-20T17:43:39.024 INFO:journalctl@ceph.mon.b.smithi079.stdout: block_align: 0 2023-07-20T17:43:39.024 INFO:journalctl@ceph.mon.b.smithi079.stdout: max_auto_readahead_size: 262144 2023-07-20T17:43:39.024 INFO:journalctl@ceph.mon.b.smithi079.stdout: prepopulate_block_cache: 0 2023-07-20T17:43:39.024 INFO:journalctl@ceph.mon.b.smithi079.stdout: initial_auto_readahead_size: 8192 2023-07-20T17:43:39.024 INFO:journalctl@ceph.mon.b.smithi079.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T17:43:39.024 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T17:43:39.024 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T17:43:39.025 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compression: NoCompression 2023-07-20T17:43:39.025 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T17:43:39.025 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T17:43:39.025 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T17:43:39.025 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.num_levels: 7 2023-07-20T17:43:39.025 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T17:43:39.025 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T17:43:39.025 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T17:43:39.026 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T17:43:39.026 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T17:43:39.026 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T17:43:39.026 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T17:43:39.026 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T17:43:39.026 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T17:43:39.026 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T17:43:39.026 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T17:43:39.027 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T17:43:39.027 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T17:43:39.027 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T17:43:39.027 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T17:43:39.027 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T17:43:39.027 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T17:43:39.027 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T17:43:39.027 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T17:43:39.028 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T17:43:39.028 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T17:43:39.028 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T17:43:39.028 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T17:43:39.028 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T17:43:39.028 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T17:43:39.028 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T17:43:39.028 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T17:43:39.029 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T17:43:39.029 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T17:43:39.029 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T17:43:39.029 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T17:43:39.029 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T17:43:39.029 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T17:43:39.029 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T17:43:39.029 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T17:43:39.030 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T17:43:39.030 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T17:43:39.030 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T17:43:39.030 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T17:43:39.030 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T17:43:39.030 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T17:43:39.030 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T17:43:39.030 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T17:43:39.031 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T17:43:39.031 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T17:43:39.031 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T17:43:39.031 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T17:43:39.031 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T17:43:39.031 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T17:43:39.031 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T17:43:39.031 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T17:43:39.032 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T17:43:39.032 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T17:43:39.032 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.table_properties_collectors: 2023-07-20T17:43:39.032 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.inplace_update_support: 0 2023-07-20T17:43:39.032 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T17:43:39.032 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T17:43:39.032 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T17:43:39.032 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T17:43:39.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.bloom_locality: 0 2023-07-20T17:43:39.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.max_successive_merges: 0 2023-07-20T17:43:39.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T17:43:39.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T17:43:39.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T17:43:39.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T17:43:39.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.ttl: 2592000 2023-07-20T17:43:39.033 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T17:43:39.034 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T17:43:39.034 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T17:43:39.034 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.enable_blob_files: false 2023-07-20T17:43:39.034 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.min_blob_size: 0 2023-07-20T17:43:39.034 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T17:43:39.034 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T17:43:39.034 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T17:43:39.035 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T17:43:39.035 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T17:43:39.035 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T17:43:39.035 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T17:43:39.035 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T17:43:39.035 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T17:43:39.035 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T17:43:39.035 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f10b75c7-1886-47e7-a3b0-268adae942ab 2023-07-20T17:43:39.036 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689875018931151, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T17:43:39.036 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T17:43:39.036 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689875018931563, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689875018, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f10b75c7-1886-47e7-a3b0-268adae942ab", "db_session_id": "0IZZA78R2OW3DCKZ199U", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T17:43:39.036 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689875018931628, "job": 1, "event": "recovery_finished"} 2023-07-20T17:43:39.036 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T17:43:39.036 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T17:43:39.036 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56297d196000 2023-07-20T17:43:39.036 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: DB pointer 0x56297d182000 2023-07-20T17:43:39.037 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T17:43:39.037 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T17:43:39.037 INFO:journalctl@ceph.mon.b.smithi079.stdout: ** DB Stats ** 2023-07-20T17:43:39.037 INFO:journalctl@ceph.mon.b.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T17:43:39.037 INFO:journalctl@ceph.mon.b.smithi079.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T17:43:39.037 INFO:journalctl@ceph.mon.b.smithi079.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T17:43:39.037 INFO:journalctl@ceph.mon.b.smithi079.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T17:43:39.038 INFO:journalctl@ceph.mon.b.smithi079.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T17:43:39.038 INFO:journalctl@ceph.mon.b.smithi079.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T17:43:39.038 INFO:journalctl@ceph.mon.b.smithi079.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T17:43:39.038 INFO:journalctl@ceph.mon.b.smithi079.stdout: 2023-07-20T17:43:39.038 INFO:journalctl@ceph.mon.b.smithi079.stdout: ** Compaction Stats [default] ** 2023-07-20T17:43:39.038 INFO:journalctl@ceph.mon.b.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T17:43:39.038 INFO:journalctl@ceph.mon.b.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T17:43:39.038 INFO:journalctl@ceph.mon.b.smithi079.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T17:43:39.039 INFO:journalctl@ceph.mon.b.smithi079.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T17:43:39.039 INFO:journalctl@ceph.mon.b.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T17:43:39.039 INFO:journalctl@ceph.mon.b.smithi079.stdout: 2023-07-20T17:43:39.039 INFO:journalctl@ceph.mon.b.smithi079.stdout: ** Compaction Stats [default] ** 2023-07-20T17:43:39.039 INFO:journalctl@ceph.mon.b.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T17:43:39.039 INFO:journalctl@ceph.mon.b.smithi079.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T17:43:39.039 INFO:journalctl@ceph.mon.b.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T17:43:39.039 INFO:journalctl@ceph.mon.b.smithi079.stdout: 2023-07-20T17:43:39.040 INFO:journalctl@ceph.mon.b.smithi079.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T17:43:39.040 INFO:journalctl@ceph.mon.b.smithi079.stdout: 2023-07-20T17:43:39.040 INFO:journalctl@ceph.mon.b.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T17:43:39.040 INFO:journalctl@ceph.mon.b.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T17:43:39.040 INFO:journalctl@ceph.mon.b.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T17:43:39.040 INFO:journalctl@ceph.mon.b.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T17:43:39.040 INFO:journalctl@ceph.mon.b.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T17:43:39.040 INFO:journalctl@ceph.mon.b.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T17:43:39.041 INFO:journalctl@ceph.mon.b.smithi079.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T17:43:39.041 INFO:journalctl@ceph.mon.b.smithi079.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T17:43:39.041 INFO:journalctl@ceph.mon.b.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T17:43:39.041 INFO:journalctl@ceph.mon.b.smithi079.stdout: Block cache BinnedLRUCache@0x56297d0fd350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-20T17:43:39.041 INFO:journalctl@ceph.mon.b.smithi079.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T17:43:39.041 INFO:journalctl@ceph.mon.b.smithi079.stdout: 2023-07-20T17:43:39.041 INFO:journalctl@ceph.mon.b.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T17:43:39.041 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-20T17:43:39.042 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: using public_addr v2:172.21.15.79:0/0 -> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] 2023-07-20T17:43:39.042 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: starting mon.b rank -1 at public addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] at bind addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:43:39.042 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(???) e0 preinit fsid 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:43:39.042 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).mds e1 new map 2023-07-20T17:43:39.042 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-20T17:43:39.042 INFO:journalctl@ceph.mon.b.smithi079.stdout: e1 2023-07-20T17:43:39.042 INFO:journalctl@ceph.mon.b.smithi079.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T17:43:39.043 INFO:journalctl@ceph.mon.b.smithi079.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T17:43:39.043 INFO:journalctl@ceph.mon.b.smithi079.stdout: legacy client fscid: -1 2023-07-20T17:43:39.043 INFO:journalctl@ceph.mon.b.smithi079.stdout: 2023-07-20T17:43:39.043 INFO:journalctl@ceph.mon.b.smithi079.stdout: No filesystems configured 2023-07-20T17:43:39.043 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T17:43:39.043 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T17:43:39.043 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T17:43:39.043 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T17:43:39.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T17:43:39.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T17:43:39.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T17:43:39.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T17:43:39.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T17:43:39.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T17:43:39.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T17:43:39.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:39.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:39.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:43:39.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Updating smithi079:/etc/ceph/ceph.conf 2023-07-20T17:43:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi097", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Updating smithi079:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Deploying cephadm binary to smithi097 2023-07-20T17:43:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Updating smithi079:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T17:43:39.046 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:39.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Added host smithi097 2023-07-20T17:43:39.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:39.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.047 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:39.048 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.048 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T17:43:39.048 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.048 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:39.048 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/199423917' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T17:43:39.048 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/199423917' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T17:43:39.048 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T17:43:39.048 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:39.049 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.049 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.049 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.049 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.049 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:43:39.049 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:39.049 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:39.049 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Updating smithi097:/etc/ceph/ceph.conf 2023-07-20T17:43:39.050 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Updating smithi097:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:39.050 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:39.050 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Updating smithi097:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T17:43:39.050 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.050 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.050 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.050 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:39.050 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.051 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:39.051 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:39.051 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:39.051 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.051 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:39.051 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:39.051 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi079:172.21.15.79=b;smithi097:172.21.15.97=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:39.051 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi079:172.21.15.79=b;smithi097:172.21.15.97=c;count:3 2023-07-20T17:43:39.052 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Deploying daemon mon.c on smithi097 2023-07-20T17:43:39.052 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:39.052 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='client.? 172.21.15.97:0/767448483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T17:43:39.052 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:43:39.052 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.a calling monitor election 2023-07-20T17:43:39.052 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:39.052 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:39.052 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:39.053 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:39.053 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.c calling monitor election 2023-07-20T17:43:39.053 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:39.053 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:39.053 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:39.053 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:39.053 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T17:43:39.053 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: monmap e2: 2 mons at {a=[v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0],c=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0]} removed_ranks: {} 2023-07-20T17:43:39.054 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: fsmap 2023-07-20T17:43:39.054 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T17:43:39.054 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mgrmap e13: x(active, since 43s) 2023-07-20T17:43:39.054 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: overall HEALTH_OK 2023-07-20T17:43:39.054 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.054 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.054 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:39.054 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:39.055 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:39.055 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: Deploying daemon mon.b on smithi079 2023-07-20T17:43:39.055 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:39.055 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:39.055 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: from='client.? 172.21.15.97:0/1213479966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T17:43:39.055 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T17:43:39.055 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-20T17:43:39.055 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta expand map: {default=false} 2023-07-20T17:43:39.056 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta from 'false' to 'false' 2023-07-20T17:43:39.056 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta expanded map: {default=false} 2023-07-20T17:43:39.056 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta expand map: {default=info} 2023-07-20T17:43:39.056 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta from 'info' to 'info' 2023-07-20T17:43:39.056 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta expanded map: {default=info} 2023-07-20T17:43:39.056 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta expand map: {default=daemon} 2023-07-20T17:43:39.056 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T17:43:39.056 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T17:43:39.057 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta expand map: {default=debug} 2023-07-20T17:43:39.057 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T17:43:39.057 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:38 smithi079 ceph-mon[98036]: expand_channel_meta expanded map: {default=debug} 2023-07-20T17:43:39.057 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:39 smithi079 systemd[1]: Started Ceph mon.b for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:43:39.406 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:43:44.074 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.071+0000 7fb2567c7700 1 -- 172.21.15.97:0/2795057017 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2500ff480 msgr2=0x7fb238005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:44.074 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.071+0000 7fb2567c7700 1 --2- 172.21.15.97:0/2795057017 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2500ff480 0x7fb238005610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb244005fa0 tx=0x7fb24400f4a0 comp rx=0 tx=0).stop 2023-07-20T17:43:44.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.072+0000 7fb2567c7700 1 -- 172.21.15.97:0/2795057017 shutdown_connections 2023-07-20T17:43:44.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.072+0000 7fb2567c7700 1 --2- 172.21.15.97:0/2795057017 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2500ff480 0x7fb238005610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:44.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.072+0000 7fb2567c7700 1 -- 172.21.15.97:0/2795057017 >> 172.21.15.97:0/2795057017 conn(0x7fb2500fad30 msgr2=0x7fb2500fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:44.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.072+0000 7fb2567c7700 1 -- 172.21.15.97:0/2795057017 shutdown_connections 2023-07-20T17:43:44.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.072+0000 7fb2567c7700 1 -- 172.21.15.97:0/2795057017 wait complete. 2023-07-20T17:43:44.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb2567c7700 1 Processor -- start 2023-07-20T17:43:44.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb2567c7700 1 -- start start 2023-07-20T17:43:44.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb2567c7700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500ff480 0x7fb25019b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:44.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb2567c7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb25019bb60 0x7fb2501956a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:44.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb2567c7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb250195be0 0x7fb250196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:44.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb2567c7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25019c1c0 con 0x7fb25019bb60 2023-07-20T17:43:44.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb2567c7700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2501a2440 con 0x7fb250195be0 2023-07-20T17:43:44.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb2567c7700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2501a25d0 con 0x7fb2500ff480 2023-07-20T17:43:44.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb24ffff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500ff480 0x7fb25019b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:44.082 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb24ffff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500ff480 0x7fb25019b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60618/0 (socket says 172.21.15.97:60618) 2023-07-20T17:43:44.082 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb24ffff700 1 -- 172.21.15.97:0/1220376718 learned_addr learned my addr 172.21.15.97:0/1220376718 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-07-20T17:43:44.082 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.073+0000 7fb24f7fe700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb25019bb60 0x7fb2501956a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:44.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24ffff700 1 -- 172.21.15.97:0/1220376718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500ff480 msgr2=0x7fb25019b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-20T17:43:44.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24ffff700 1 -- 172.21.15.97:0/1220376718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500ff480 msgr2=0x7fb25019b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T17:43:44.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24ffff700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500ff480 0x7fb25019b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T17:43:44.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24ffff700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500ff480 0x7fb25019b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T17:43:44.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb254d64700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb250195be0 0x7fb250196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:44.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb254d64700 1 -- 172.21.15.97:0/1220376718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb250195be0 msgr2=0x7fb250196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-20T17:43:44.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb254d64700 1 -- 172.21.15.97:0/1220376718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb250195be0 msgr2=0x7fb250196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T17:43:44.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb254d64700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb250195be0 0x7fb250196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T17:43:44.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb254d64700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb250195be0 0x7fb250196090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T17:43:44.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24f7fe700 1 -- 172.21.15.97:0/1220376718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500ff480 msgr2=0x7fb25019b620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T17:43:44.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24f7fe700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500ff480 0x7fb25019b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:44.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24f7fe700 1 -- 172.21.15.97:0/1220376718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb250195be0 msgr2=0x7fb250196090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T17:43:44.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24f7fe700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb250195be0 0x7fb250196090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:44.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24f7fe700 1 -- 172.21.15.97:0/1220376718 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2440139a0 con 0x7fb25019bb60 2023-07-20T17:43:44.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24f7fe700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb25019bb60 0x7fb2501956a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb24000b550 tx=0x7fb24000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:44.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24d7fa700 1 -- 172.21.15.97:0/1220376718 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240004230 con 0x7fb25019bb60 2023-07-20T17:43:44.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb2567c7700 1 -- 172.21.15.97:0/1220376718 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2501a28b0 con 0x7fb25019bb60 2023-07-20T17:43:44.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24d7fa700 1 -- 172.21.15.97:0/1220376718 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb240004390 con 0x7fb25019bb60 2023-07-20T17:43:44.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb24d7fa700 1 -- 172.21.15.97:0/1220376718 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240011af0 con 0x7fb25019bb60 2023-07-20T17:43:44.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.074+0000 7fb2567c7700 1 -- 172.21.15.97:0/1220376718 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2501a2db0 con 0x7fb25019bb60 2023-07-20T17:43:44.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.075+0000 7fb24d7fa700 1 -- 172.21.15.97:0/1220376718 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fb240011c50 con 0x7fb25019bb60 2023-07-20T17:43:44.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.075+0000 7fb24d7fa700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fb23c038c80 0x7fb23c03b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:44.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.076+0000 7fb24d7fa700 1 -- 172.21.15.97:0/1220376718 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb24004bd80 con 0x7fb25019bb60 2023-07-20T17:43:44.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.076+0000 7fb24ffff700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fb23c038c80 0x7fb23c03b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:44.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.076+0000 7fb2567c7700 1 -- 172.21.15.97:0/1220376718 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb250061da0 con 0x7fb25019bb60 2023-07-20T17:43:44.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.076+0000 7fb24ffff700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fb23c038c80 0x7fb23c03b140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb24400f470 tx=0x7fb24400f3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:44.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.082+0000 7fb24d7fa700 1 -- 172.21.15.97:0/1220376718 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb24001a400 con 0x7fb25019bb60 2023-07-20T17:43:44.355 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.354+0000 7fb2567c7700 1 -- 172.21.15.97:0/1220376718 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2501a3030 con 0x7fb25019bb60 2023-07-20T17:43:44.356 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.355+0000 7fb24d7fa700 1 -- 172.21.15.97:0/1220376718 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1268 (secure 0 0 0) 0x7fb240049020 con 0x7fb25019bb60 2023-07-20T17:43:44.356 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-07-20T17:43:44.356 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":3,"fsid":"8c050cea-2724-11ee-9b34-001a4aab830c","modified":"2023-07-20T17:43:38.963672Z","created":"2023-07-20T17:41:55.142063Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-07-20T17:43:44.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 -- 172.21.15.97:0/1220376718 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fb23c038c80 msgr2=0x7fb23c03b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:44.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fb23c038c80 0x7fb23c03b140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb24400f470 tx=0x7fb24400f3c0 comp rx=0 tx=0).stop 2023-07-20T17:43:44.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 -- 172.21.15.97:0/1220376718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb25019bb60 msgr2=0x7fb2501956a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:44.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb25019bb60 0x7fb2501956a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb24000b550 tx=0x7fb24000b910 comp rx=0 tx=0).stop 2023-07-20T17:43:44.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 -- 172.21.15.97:0/1220376718 shutdown_connections 2023-07-20T17:43:44.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fb23c038c80 0x7fb23c03b140 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:44.360 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2500ff480 0x7fb25019b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:44.360 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb250195be0 0x7fb250196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:44.360 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 --2- 172.21.15.97:0/1220376718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb25019bb60 0x7fb2501956a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:44.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 -- 172.21.15.97:0/1220376718 >> 172.21.15.97:0/1220376718 conn(0x7fb2500fad30 msgr2=0x7fb250103060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:44.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 -- 172.21.15.97:0/1220376718 shutdown_connections 2023-07-20T17:43:44.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:44.357+0000 7fb2567c7700 1 -- 172.21.15.97:0/1220376718 wait complete. 2023-07-20T17:43:44.362 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 3 2023-07-20T17:43:44.382 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: mon.a calling monitor election 2023-07-20T17:43:44.383 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: mon.c calling monitor election 2023-07-20T17:43:44.383 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:43:44.383 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.383 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:44.384 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.384 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:44.384 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.384 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: mon.b calling monitor election 2023-07-20T17:43:44.384 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.384 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:44.385 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.385 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.385 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T17:43:44.385 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: monmap e3: 3 mons at {a=[v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0],b=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],c=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0]} removed_ranks: {} 2023-07-20T17:43:44.385 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: fsmap 2023-07-20T17:43:44.385 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T17:43:44.386 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: mgrmap e13: x(active, since 51s) 2023-07-20T17:43:44.386 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: overall HEALTH_OK 2023-07-20T17:43:44.386 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:44.386 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:44.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: mon.a calling monitor election 2023-07-20T17:43:44.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: mon.c calling monitor election 2023-07-20T17:43:44.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:43:44.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:44.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:44.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: mon.b calling monitor election 2023-07-20T17:43:44.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:44.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T17:43:44.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: monmap e3: 3 mons at {a=[v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0],b=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],c=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0]} removed_ranks: {} 2023-07-20T17:43:44.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: fsmap 2023-07-20T17:43:44.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T17:43:44.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: mgrmap e13: x(active, since 51s) 2023-07-20T17:43:44.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: overall HEALTH_OK 2023-07-20T17:43:44.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:44.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:44.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: mon.a calling monitor election 2023-07-20T17:43:44.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: mon.c calling monitor election 2023-07-20T17:43:44.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:43:44.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:43:44.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:44.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: mon.b calling monitor election 2023-07-20T17:43:44.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:44.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T17:43:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: monmap e3: 3 mons at {a=[v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0],b=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],c=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0]} removed_ranks: {} 2023-07-20T17:43:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: fsmap 2023-07-20T17:43:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T17:43:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: mgrmap e13: x(active, since 51s) 2023-07-20T17:43:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: overall HEALTH_OK 2023-07-20T17:43:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:44.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:44.774 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-20T17:43:44.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph config generate-minimal-conf 2023-07-20T17:43:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:43:45.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:45 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:45.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:45 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:45.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:45 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:45.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:45 smithi049 ceph-mon[96530]: from='client.? 172.21.15.97:0/1220376718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T17:43:45.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:45 smithi049 ceph-mon[96530]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:45.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:45 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:45.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:45 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:45.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:45 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:45.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:45 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:45.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:45 smithi097 ceph-mon[98398]: from='client.? 172.21.15.97:0/1220376718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T17:43:45.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:45 smithi097 ceph-mon[98398]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:45.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:45 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:45.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:45 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:45.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:45 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:45.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:45 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:45.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:45 smithi079 ceph-mon[98036]: from='client.? 172.21.15.97:0/1220376718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T17:43:45.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:45 smithi079 ceph-mon[98036]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:45.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:45 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:43:46.170 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:43:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:43:45.962+0000 7f0405ab6700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-07-20T17:43:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.709+0000 7fc9c5afe700 1 -- 172.21.15.49:0/3372783896 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9c0101440 msgr2=0x7fc9c0101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:46.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.709+0000 7fc9c5afe700 1 --2- 172.21.15.49:0/3372783896 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9c0101440 0x7fc9c0101820 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0009910 tx=0x7fc9b0009c20 comp rx=0 tx=0).stop 2023-07-20T17:43:46.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.709+0000 7fc9c5afe700 1 -- 172.21.15.49:0/3372783896 shutdown_connections 2023-07-20T17:43:46.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.709+0000 7fc9c5afe700 1 --2- 172.21.15.49:0/3372783896 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9c0101440 0x7fc9c0101820 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:46.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.709+0000 7fc9c5afe700 1 -- 172.21.15.49:0/3372783896 >> 172.21.15.49:0/3372783896 conn(0x7fc9c00fabf0 msgr2=0x7fc9c00fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:46.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.710+0000 7fc9c5afe700 1 -- 172.21.15.49:0/3372783896 shutdown_connections 2023-07-20T17:43:46.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.710+0000 7fc9c5afe700 1 -- 172.21.15.49:0/3372783896 wait complete. 2023-07-20T17:43:46.713 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.710+0000 7fc9c5afe700 1 Processor -- start 2023-07-20T17:43:46.713 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9c5afe700 1 -- start start 2023-07-20T17:43:46.713 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9c5afe700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9c0101440 0x7fc9c019b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:46.713 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9c5afe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9c019bb70 0x7fc9c01956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:46.713 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9c5afe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9c0195bf0 0x7fc9c0196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:46.713 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9c5afe700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c019c0b0 con 0x7fc9c019bb70 2023-07-20T17:43:46.713 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9c5afe700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c019c220 con 0x7fc9c0195bf0 2023-07-20T17:43:46.714 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9c5afe700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c01a4850 con 0x7fc9c0101440 2023-07-20T17:43:46.714 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9beffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9c019bb70 0x7fc9c01956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:46.714 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9beffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9c019bb70 0x7fc9c01956b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35562/0 (socket says 172.21.15.49:35562) 2023-07-20T17:43:46.714 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9beffd700 1 -- 172.21.15.49:0/2977393450 learned_addr learned my addr 172.21.15.49:0/2977393450 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:46.714 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9bf7fe700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9c0101440 0x7fc9c019b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:46.714 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.711+0000 7fc9bffff700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9c0195bf0 0x7fc9c0196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:46.714 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9beffd700 1 -- 172.21.15.49:0/2977393450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9c0101440 msgr2=0x7fc9c019b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:46.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9beffd700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9c0101440 0x7fc9c019b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:46.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9beffd700 1 -- 172.21.15.49:0/2977393450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9c0195bf0 msgr2=0x7fc9c0196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:46.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9beffd700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9c0195bf0 0x7fc9c0196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:46.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9beffd700 1 -- 172.21.15.49:0/2977393450 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b00095f0 con 0x7fc9c019bb70 2023-07-20T17:43:46.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9bf7fe700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9c0101440 0x7fc9c019b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T17:43:46.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9bffff700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9c0195bf0 0x7fc9c0196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T17:43:46.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9beffd700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9c019bb70 0x7fc9c01956b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400b780 tx=0x7fc9b400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:46.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9bcff9700 1 -- 172.21.15.49:0/2977393450 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4004230 con 0x7fc9c019bb70 2023-07-20T17:43:46.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9bcff9700 1 -- 172.21.15.49:0/2977393450 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9b4004390 con 0x7fc9c019bb70 2023-07-20T17:43:46.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9c5afe700 1 -- 172.21.15.49:0/2977393450 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c01a0560 con 0x7fc9c019bb70 2023-07-20T17:43:46.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9bcff9700 1 -- 172.21.15.49:0/2977393450 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4011b90 con 0x7fc9c019bb70 2023-07-20T17:43:46.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.712+0000 7fc9c5afe700 1 -- 172.21.15.49:0/2977393450 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c01a0ab0 con 0x7fc9c019bb70 2023-07-20T17:43:46.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.713+0000 7fc9c5afe700 1 -- 172.21.15.49:0/2977393450 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c0107950 con 0x7fc9c019bb70 2023-07-20T17:43:46.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.713+0000 7fc9bcff9700 1 -- 172.21.15.49:0/2977393450 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fc9b4011cf0 con 0x7fc9c019bb70 2023-07-20T17:43:46.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.713+0000 7fc9bcff9700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9a8038c30 0x7fc9a803b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:46.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.713+0000 7fc9bcff9700 1 -- 172.21.15.49:0/2977393450 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc9b404a0d0 con 0x7fc9c019bb70 2023-07-20T17:43:46.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.713+0000 7fc9bf7fe700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9a8038c30 0x7fc9a803b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:46.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.714+0000 7fc9bf7fe700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9a8038c30 0x7fc9a803b0f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0006010 tx=0x7fc9b000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:46.719 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.718+0000 7fc9bcff9700 1 -- 172.21.15.49:0/2977393450 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc9b4018d20 con 0x7fc9c019bb70 2023-07-20T17:43:46.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.919+0000 7fc9c5afe700 1 -- 172.21.15.49:0/2977393450 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fc9c002c970 con 0x7fc9c019bb70 2023-07-20T17:43:46.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.920+0000 7fc9bcff9700 1 -- 172.21.15.49:0/2977393450 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+271 (secure 0 0 0) 0x7fc9b4016070 con 0x7fc9c019bb70 2023-07-20T17:43:46.921 INFO:teuthology.orchestra.run.smithi049.stdout:# minimal ceph.conf for 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:43:46.921 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2023-07-20T17:43:46.921 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:43:46.921 INFO:teuthology.orchestra.run.smithi049.stdout: mon_host = [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-07-20T17:43:46.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 -- 172.21.15.49:0/2977393450 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9a8038c30 msgr2=0x7fc9a803b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:46.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9a8038c30 0x7fc9a803b0f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0006010 tx=0x7fc9b000b540 comp rx=0 tx=0).stop 2023-07-20T17:43:46.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 -- 172.21.15.49:0/2977393450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9c019bb70 msgr2=0x7fc9c01956b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:46.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9c019bb70 0x7fc9c01956b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400b780 tx=0x7fc9b400ba90 comp rx=0 tx=0).stop 2023-07-20T17:43:46.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 -- 172.21.15.49:0/2977393450 shutdown_connections 2023-07-20T17:43:46.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9a8038c30 0x7fc9a803b0f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:46.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9c0101440 0x7fc9c019b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:46.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9c0195bf0 0x7fc9c0196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:46.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 --2- 172.21.15.49:0/2977393450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9c019bb70 0x7fc9c01956b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:46.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 -- 172.21.15.49:0/2977393450 >> 172.21.15.49:0/2977393450 conn(0x7fc9c00fabf0 msgr2=0x7fc9c0104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:46.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 -- 172.21.15.49:0/2977393450 shutdown_connections 2023-07-20T17:43:46.925 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:46.922+0000 7fc9c5afe700 1 -- 172.21.15.49:0/2977393450 wait complete. 2023-07-20T17:43:46.949 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:47.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:47.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:47.369 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-20T17:43:47.369 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:43:47.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T17:43:47.398 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:43:47.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:47.470 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-07-20T17:43:47.471 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T17:43:47.507 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-07-20T17:43:47.508 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:47.578 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-07-20T17:43:47.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T17:43:47.615 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-07-20T17:43:47.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T17:43:47.685 INFO:tasks.cephadm:Adding mgr.x on smithi049 2023-07-20T17:43:47.685 INFO:tasks.cephadm:Adding mgr.y on smithi079 2023-07-20T17:43:47.686 INFO:tasks.cephadm:Adding mgr.z on smithi097 2023-07-20T17:43:47.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch apply mgr '3;smithi049=x;smithi079=y;smithi097=z' 2023-07-20T17:43:47.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: Updating smithi049:/etc/ceph/ceph.conf 2023-07-20T17:43:47.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: Updating smithi079:/etc/ceph/ceph.conf 2023-07-20T17:43:47.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: Updating smithi097:/etc/ceph/ceph.conf 2023-07-20T17:43:47.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: Updating smithi049:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:47.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: Updating smithi079:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:47.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: Updating smithi097:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:47.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T17:43:47.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:47.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T17:43:47.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:47.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: Reconfiguring daemon mon.a on smithi049 2023-07-20T17:43:47.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:47.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:47 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/2977393450' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:47.835 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: Updating smithi049:/etc/ceph/ceph.conf 2023-07-20T17:43:47.835 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: Updating smithi079:/etc/ceph/ceph.conf 2023-07-20T17:43:47.835 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: Updating smithi097:/etc/ceph/ceph.conf 2023-07-20T17:43:47.835 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: Updating smithi049:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:47.835 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: Updating smithi079:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:47.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: Updating smithi097:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:47.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T17:43:47.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:47.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T17:43:47.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:47.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: Reconfiguring daemon mon.a on smithi049 2023-07-20T17:43:47.837 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:47.837 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:47 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2977393450' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:48.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: Updating smithi049:/etc/ceph/ceph.conf 2023-07-20T17:43:48.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: Updating smithi079:/etc/ceph/ceph.conf 2023-07-20T17:43:48.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: Updating smithi097:/etc/ceph/ceph.conf 2023-07-20T17:43:48.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: Updating smithi049:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:48.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: Updating smithi079:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:48.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: Updating smithi097:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T17:43:48.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T17:43:48.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:48.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T17:43:48.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:48.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: Reconfiguring daemon mon.a on smithi049 2023-07-20T17:43:48.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:48.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:47 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/2977393450' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:48.276 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:43:48.952 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:48 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:48.952 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:48 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:48.952 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:48 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:48.953 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:48 smithi079 ceph-mon[98036]: Reconfiguring mon.b (monmap changed)... 2023-07-20T17:43:48.953 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:48 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:48.953 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:48 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T17:43:48.953 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:48 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:48.953 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:48 smithi079 ceph-mon[98036]: Reconfiguring daemon mon.b on smithi079 2023-07-20T17:43:48.953 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:48 smithi079 ceph-mon[98036]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:49.139 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:48 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:49.139 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:48 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:49.139 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:48 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:49.140 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:48 smithi097 ceph-mon[98398]: Reconfiguring mon.b (monmap changed)... 2023-07-20T17:43:49.140 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:48 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:49.140 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:48 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T17:43:49.140 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:48 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:49.140 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:48 smithi097 ceph-mon[98398]: Reconfiguring daemon mon.b on smithi079 2023-07-20T17:43:49.140 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:48 smithi097 ceph-mon[98398]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:49.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:48 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:49.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:48 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:49.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:48 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:49.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:48 smithi049 ceph-mon[96530]: Reconfiguring mon.b (monmap changed)... 2023-07-20T17:43:49.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:48 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:49.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:48 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T17:43:49.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:48 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:49.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:48 smithi049 ceph-mon[96530]: Reconfiguring daemon mon.b on smithi079 2023-07-20T17:43:49.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:48 smithi049 ceph-mon[96530]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:49.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.600+0000 7f556dfce700 1 -- 172.21.15.97:0/965485552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55680ff290 msgr2=0x7f55680ff6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:49.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.600+0000 7f556dfce700 1 --2- 172.21.15.97:0/965485552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55680ff290 0x7f55680ff6b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f555800b0a0 tx=0x7f555800b3b0 comp rx=0 tx=0).stop 2023-07-20T17:43:49.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.600+0000 7f556dfce700 1 -- 172.21.15.97:0/965485552 shutdown_connections 2023-07-20T17:43:49.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.600+0000 7f556dfce700 1 --2- 172.21.15.97:0/965485552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55680ff290 0x7f55680ff6b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:49.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.600+0000 7f556dfce700 1 -- 172.21.15.97:0/965485552 >> 172.21.15.97:0/965485552 conn(0x7f55680fa810 msgr2=0x7f55680fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:49.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.600+0000 7f556dfce700 1 -- 172.21.15.97:0/965485552 shutdown_connections 2023-07-20T17:43:49.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.600+0000 7f556dfce700 1 -- 172.21.15.97:0/965485552 wait complete. 2023-07-20T17:43:49.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f556dfce700 1 Processor -- start 2023-07-20T17:43:49.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f556dfce700 1 -- start start 2023-07-20T17:43:49.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f556dfce700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55680ff290 0x7f5568195720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:49.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f556dfce700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568195c60 0x7f556819acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:49.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f556dfce700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55681960e0 0x7f556819d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:49.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f556dfce700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556819da00 con 0x7f55680ff290 2023-07-20T17:43:49.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f556dfce700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556819db70 con 0x7f55681960e0 2023-07-20T17:43:49.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f556dfce700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556819dd30 con 0x7f5568195c60 2023-07-20T17:43:49.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f5566ffd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568195c60 0x7f556819acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:49.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f5566ffd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568195c60 0x7f556819acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60630/0 (socket says 172.21.15.97:60630) 2023-07-20T17:43:49.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f5566ffd700 1 -- 172.21.15.97:0/1751450934 learned_addr learned my addr 172.21.15.97:0/1751450934 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-07-20T17:43:49.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f55677fe700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55680ff290 0x7f5568195720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:49.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.601+0000 7f5567fff700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55681960e0 0x7f556819d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:49.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f5566ffd700 1 -- 172.21.15.97:0/1751450934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55681960e0 msgr2=0x7f556819d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:49.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f5566ffd700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55681960e0 0x7f556819d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:49.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f5566ffd700 1 -- 172.21.15.97:0/1751450934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55680ff290 msgr2=0x7f5568195720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:49.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f5566ffd700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55680ff290 0x7f5568195720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:49.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f5566ffd700 1 -- 172.21.15.97:0/1751450934 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5558009d00 con 0x7f5568195c60 2023-07-20T17:43:49.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f55677fe700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55680ff290 0x7f5568195720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:43:49.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f5566ffd700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568195c60 0x7f556819acd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f555c00b780 tx=0x7f555c00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:49.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f5564ff9700 1 -- 172.21.15.97:0/1751450934 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c004230 con 0x7f5568195c60 2023-07-20T17:43:49.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f5564ff9700 1 -- 172.21.15.97:0/1751450934 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f555c004390 con 0x7f5568195c60 2023-07-20T17:43:49.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f556dfce700 1 -- 172.21.15.97:0/1751450934 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556819dfb0 con 0x7f5568195c60 2023-07-20T17:43:49.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f5564ff9700 1 -- 172.21.15.97:0/1751450934 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c011ad0 con 0x7f5568195c60 2023-07-20T17:43:49.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.602+0000 7f556dfce700 1 -- 172.21.15.97:0/1751450934 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556819e550 con 0x7f5568195c60 2023-07-20T17:43:49.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.603+0000 7f556dfce700 1 -- 172.21.15.97:0/1751450934 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556818ee60 con 0x7f5568195c60 2023-07-20T17:43:49.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.603+0000 7f5564ff9700 1 -- 172.21.15.97:0/1751450934 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f555c011c30 con 0x7f5568195c60 2023-07-20T17:43:49.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.603+0000 7f5564ff9700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5550038c30 0x7f555003b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:49.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.603+0000 7f5564ff9700 1 -- 172.21.15.97:0/1751450934 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f555c04ac60 con 0x7f5568195c60 2023-07-20T17:43:49.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.604+0000 7f55677fe700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5550038c30 0x7f555003b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:49.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.604+0000 7f55677fe700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5550038c30 0x7f555003b0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f55580062a0 tx=0x7f5558006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:49.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.608+0000 7f5564ff9700 1 -- 172.21.15.97:0/1751450934 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f555c01ae40 con 0x7f5568195c60 2023-07-20T17:43:49.825 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.825+0000 7f556dfce700 1 -- 172.21.15.97:0/1751450934 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi049=x;smithi079=y;smithi097=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f5568060c40 con 0x7f5550038c30 2023-07-20T17:43:49.832 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.832+0000 7f5564ff9700 1 -- 172.21.15.97:0/1751450934 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5568060c40 con 0x7f5550038c30 2023-07-20T17:43:49.832 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled mgr update... 2023-07-20T17:43:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 -- 172.21.15.97:0/1751450934 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5550038c30 msgr2=0x7f555003b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5550038c30 0x7f555003b0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f55580062a0 tx=0x7f5558006210 comp rx=0 tx=0).stop 2023-07-20T17:43:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 -- 172.21.15.97:0/1751450934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568195c60 msgr2=0x7f556819acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568195c60 0x7f556819acd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f555c00b780 tx=0x7f555c00ba90 comp rx=0 tx=0).stop 2023-07-20T17:43:49.834 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 -- 172.21.15.97:0/1751450934 shutdown_connections 2023-07-20T17:43:49.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f5550038c30 0x7f555003b0f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:49.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55681960e0 0x7f556819d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:49.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55680ff290 0x7f5568195720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:49.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 --2- 172.21.15.97:0/1751450934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568195c60 0x7f556819acd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:49.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 -- 172.21.15.97:0/1751450934 >> 172.21.15.97:0/1751450934 conn(0x7f55680fa810 msgr2=0x7f55680fb650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:49.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 -- 172.21.15.97:0/1751450934 shutdown_connections 2023-07-20T17:43:49.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:43:49.834+0000 7f556dfce700 1 -- 172.21.15.97:0/1751450934 wait complete. 2023-07-20T17:43:50.327 DEBUG:teuthology.orchestra.run.smithi079:mgr.y> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.y.service 2023-07-20T17:43:50.330 DEBUG:teuthology.orchestra.run.smithi097:mgr.z> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.z.service 2023-07-20T17:43:50.333 INFO:tasks.cephadm:Deploying OSDs... 2023-07-20T17:43:50.334 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-07-20T17:43:50.334 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T17:43:50.357 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T17:43:50.357 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2023-07-20T17:43:50.361 INFO:journalctl@ceph.mgr.z.smithi097.stdout:-- Logs begin at Thu 2023-07-20 17:15:39 UTC. -- 2023-07-20T17:43:50.366 INFO:journalctl@ceph.mgr.y.smithi079.stdout:-- Logs begin at Thu 2023-07-20 17:15:42 UTC. -- 2023-07-20T17:43:50.418 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T17:43:50.418 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:50.419 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 382020 Links: 1 2023-07-20T17:43:50.419 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:50.419 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:50.419 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-07-20 17:43:03.942874476 +0000 2023-07-20T17:43:50.419 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-07-20 17:26:55.606297321 +0000 2023-07-20T17:43:50.419 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-07-20 17:26:55.606297321 +0000 2023-07-20T17:43:50.419 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-07-20T17:43:50.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T17:43:50.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:50.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:50.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:50 smithi049 ceph-mon[96530]: Reconfiguring mon.c (monmap changed)... 2023-07-20T17:43:50.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:50.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T17:43:50.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:50.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:50 smithi049 ceph-mon[96530]: Reconfiguring daemon mon.c on smithi097 2023-07-20T17:43:50.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:50.491 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-07-20T17:43:50.492 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-07-20T17:43:50.492 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000123167 s, 4.2 MB/s 2023-07-20T17:43:50.494 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T17:43:50.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:50.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:50.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:50 smithi097 ceph-mon[98398]: Reconfiguring mon.c (monmap changed)... 2023-07-20T17:43:50.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:50.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T17:43:50.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:50.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:50 smithi097 ceph-mon[98398]: Reconfiguring daemon mon.c on smithi097 2023-07-20T17:43:50.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:50.556 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2023-07-20T17:43:50.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:50.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:50.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:50 smithi079 ceph-mon[98036]: Reconfiguring mon.c (monmap changed)... 2023-07-20T17:43:50.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T17:43:50.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T17:43:50.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:50.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:50 smithi079 ceph-mon[98036]: Reconfiguring daemon mon.c on smithi097 2023-07-20T17:43:50.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:50.617 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T17:43:50.617 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:50.617 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 367581 Links: 1 2023-07-20T17:43:50.617 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:50.618 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:50.618 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-07-20 17:43:03.942874476 +0000 2023-07-20T17:43:50.618 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-07-20 17:26:56.136282341 +0000 2023-07-20T17:43:50.618 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-07-20 17:26:56.136282341 +0000 2023-07-20T17:43:50.618 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-07-20T17:43:50.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T17:43:50.688 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-07-20T17:43:50.688 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-07-20T17:43:50.688 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000131455 s, 3.9 MB/s 2023-07-20T17:43:50.690 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T17:43:50.751 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2023-07-20T17:43:50.810 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T17:43:50.810 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:50.810 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 382202 Links: 1 2023-07-20T17:43:50.810 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:50.811 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:50.811 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-07-20 17:43:03.942874476 +0000 2023-07-20T17:43:50.811 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-07-20 17:26:56.677267051 +0000 2023-07-20T17:43:50.811 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-07-20 17:26:56.677267051 +0000 2023-07-20T17:43:50.811 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-07-20T17:43:50.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T17:43:50.878 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-07-20T17:43:50.879 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-07-20T17:43:50.879 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000128665 s, 4.0 MB/s 2023-07-20T17:43:50.880 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T17:43:50.938 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2023-07-20T17:43:50.996 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T17:43:50.997 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:50.997 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 376519 Links: 1 2023-07-20T17:43:50.997 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:50.997 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:50.998 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-07-20 17:43:03.942874476 +0000 2023-07-20T17:43:50.998 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-07-20 17:26:57.217251789 +0000 2023-07-20T17:43:50.998 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-07-20 17:26:57.217251789 +0000 2023-07-20T17:43:50.998 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-07-20T17:43:50.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T17:43:51.064 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-07-20T17:43:51.064 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-07-20T17:43:51.064 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000118858 s, 4.3 MB/s 2023-07-20T17:43:51.066 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T17:43:51.124 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-07-20T17:43:51.124 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T17:43:51.146 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T17:43:51.146 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2023-07-20T17:43:51.206 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T17:43:51.207 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:51.207 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 380116 Links: 1 2023-07-20T17:43:51.207 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:51.207 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:51.207 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2023-07-20 17:43:38.374762609 +0000 2023-07-20T17:43:51.207 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2023-07-20 17:26:11.642353437 +0000 2023-07-20T17:43:51.208 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2023-07-20 17:26:11.642353437 +0000 2023-07-20T17:43:51.208 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2023-07-20T17:43:51.208 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T17:43:51.275 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2023-07-20T17:43:51.276 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2023-07-20T17:43:51.276 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000248507 s, 2.1 MB/s 2023-07-20T17:43:51.277 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T17:43:51.337 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2023-07-20T17:43:51.395 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T17:43:51.395 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:51.396 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 374260 Links: 1 2023-07-20T17:43:51.396 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:51.396 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:51.396 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2023-07-20 17:43:38.374762609 +0000 2023-07-20T17:43:51.396 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2023-07-20 17:26:12.161341187 +0000 2023-07-20T17:43:51.396 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2023-07-20 17:26:12.161341187 +0000 2023-07-20T17:43:51.396 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2023-07-20T17:43:51.397 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T17:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2023-07-20T17:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2023-07-20T17:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000230261 s, 2.2 MB/s 2023-07-20T17:43:51.464 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T17:43:51.524 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2023-07-20T17:43:51.582 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T17:43:51.582 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:51.582 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 380267 Links: 1 2023-07-20T17:43:51.583 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:51.583 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:51.583 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2023-07-20 17:43:38.374762609 +0000 2023-07-20T17:43:51.583 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2023-07-20 17:26:12.679328960 +0000 2023-07-20T17:43:51.583 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2023-07-20 17:26:12.679328960 +0000 2023-07-20T17:43:51.583 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2023-07-20T17:43:51.583 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T17:43:51.648 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2023-07-20T17:43:51.649 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2023-07-20T17:43:51.649 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000205629 s, 2.5 MB/s 2023-07-20T17:43:51.650 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T17:43:51.708 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2023-07-20T17:43:51.765 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T17:43:51.766 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:51.766 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 381444 Links: 1 2023-07-20T17:43:51.766 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:51.766 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:51.766 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2023-07-20 17:43:38.374762609 +0000 2023-07-20T17:43:51.766 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2023-07-20 17:26:13.207316498 +0000 2023-07-20T17:43:51.766 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2023-07-20 17:26:13.207316498 +0000 2023-07-20T17:43:51.767 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2023-07-20T17:43:51.767 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T17:43:51.832 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2023-07-20T17:43:51.832 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2023-07-20T17:43:51.832 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000228315 s, 2.2 MB/s 2023-07-20T17:43:51.833 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T17:43:51.891 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-07-20T17:43:51.892 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T17:43:51.909 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T17:43:51.910 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2023-07-20T17:43:51.967 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T17:43:51.967 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:51.967 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 377623 Links: 1 2023-07-20T17:43:51.968 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:51.968 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:51.968 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-07-20 17:43:30.062837172 +0000 2023-07-20T17:43:51.968 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-07-20 17:29:01.018117160 +0000 2023-07-20T17:43:51.968 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-07-20 17:29:01.018117160 +0000 2023-07-20T17:43:51.968 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-07-20T17:43:51.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T17:43:51.977 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi049=x;smithi079=y;smithi097=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:51.978 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: Saving service mgr spec with placement smithi049=x;smithi079=y;smithi097=z;count:3 2023-07-20T17:43:51.978 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:51.978 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:51.978 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:51.978 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:51.978 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:51.979 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:51.979 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T17:43:51.979 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:51.979 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T17:43:51.979 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T17:43:51.979 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:51 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:51.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi049=x;smithi079=y;smithi097=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:51.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: Saving service mgr spec with placement smithi049=x;smithi079=y;smithi097=z;count:3 2023-07-20T17:43:51.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:51.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:51.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:51.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:51.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:51.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:51.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T17:43:51.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:51.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T17:43:51.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T17:43:51.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:51 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:52.008 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-07-20T17:43:52.009 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-07-20T17:43:52.009 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.00242676 s, 211 kB/s 2023-07-20T17:43:52.010 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T17:43:52.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi049=x;smithi079=y;smithi097=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:52.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: Saving service mgr spec with placement smithi049=x;smithi079=y;smithi097=z;count:3 2023-07-20T17:43:52.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:52.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:52.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:52.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:52.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:52.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:52.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T17:43:52.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:52.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T17:43:52.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T17:43:52.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:51 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:52.068 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2023-07-20T17:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T17:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:52.127 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 378536 Links: 1 2023-07-20T17:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-07-20 17:43:30.062837172 +0000 2023-07-20T17:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-07-20 17:29:01.536108166 +0000 2023-07-20T17:43:52.128 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-07-20 17:29:01.536108166 +0000 2023-07-20T17:43:52.129 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-07-20T17:43:52.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T17:43:52.200 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-07-20T17:43:52.200 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-07-20T17:43:52.200 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000208643 s, 2.5 MB/s 2023-07-20T17:43:52.201 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T17:43:52.260 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2023-07-20T17:43:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T17:43:52.317 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:52.317 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 378837 Links: 1 2023-07-20T17:43:52.318 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:52.318 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:52.318 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-07-20 17:43:52.263430146 +0000 2023-07-20T17:43:52.318 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-07-20 17:29:02.095098461 +0000 2023-07-20T17:43:52.318 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-07-20 17:29:02.095098461 +0000 2023-07-20T17:43:52.318 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-07-20T17:43:52.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T17:43:52.386 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-07-20T17:43:52.387 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-07-20T17:43:52.387 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.00020711 s, 2.5 MB/s 2023-07-20T17:43:52.388 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T17:43:52.446 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2023-07-20T17:43:52.504 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T17:43:52.505 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T17:43:52.505 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 6h/6d Inode: 375696 Links: 1 2023-07-20T17:43:52.505 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T17:43:52.505 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T17:43:52.505 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-07-20 17:43:52.263430146 +0000 2023-07-20T17:43:52.506 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-07-20 17:29:02.621089328 +0000 2023-07-20T17:43:52.506 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-07-20 17:29:02.621089328 +0000 2023-07-20T17:43:52.506 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-07-20T17:43:52.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T17:43:52.568 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:52 smithi097 systemd[1]: Starting Ceph mgr.z for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-07-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-07-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000185091 s, 2.8 MB/s 2023-07-20T17:43:52.572 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T17:43:52.630 INFO:tasks.cephadm:Deploying osd.0 on smithi049 with /dev/vg_nvme/lv_4... 2023-07-20T17:43:52.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T17:43:52.937 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:52 smithi097 ceph-mon[98398]: Deploying daemon mgr.z on smithi097 2023-07-20T17:43:52.938 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:52 smithi097 podman[100453]: 2023-07-20T17:43:52.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:52 smithi049 ceph-mon[96530]: Deploying daemon mgr.z on smithi097 2023-07-20T17:43:53.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:52 smithi079 ceph-mon[98036]: Deploying daemon mgr.z on smithi097 2023-07-20T17:43:53.205 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:43:53.228 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:52 smithi097 bash[100453]: 4909aa7bc0066787a4a45b744e79993bb8b01d0db453eba348eed415105f9f6e 2023-07-20T17:43:53.228 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:52 smithi097 systemd[1]: Started Ceph mgr.z for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:43:53.229 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:53.059+0000 7f169628c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T17:43:53.229 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:53.139+0000 7f169628c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T17:43:53.229 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:53.228+0000 7f169628c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T17:43:53.490 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:53.321+0000 7f169628c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T17:43:53.762 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:53.489+0000 7f169628c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T17:43:53.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:53 smithi049 ceph-mon[96530]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:53.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:53.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:53.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:53.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T17:43:53.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T17:43:53.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T17:43:53.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:54.001 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:53 smithi079 ceph-mon[98036]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:54.001 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:54.001 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:54.002 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:54.002 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T17:43:54.002 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T17:43:54.002 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T17:43:54.002 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:54.016 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:53.794+0000 7f169628c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T17:43:54.016 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:53.881+0000 7f169628c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T17:43:54.016 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:54 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:54.007+0000 7f169628c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T17:43:54.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-mon[98398]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:54.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:54.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:54.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:54.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T17:43:54.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T17:43:54.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T17:43:54.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:54.516 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:54 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:54.101+0000 7f169628c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T17:43:54.670 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T17:43:54.670 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T17:43:54.670 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2023-07-20T17:43:54.670 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2023-07-20T17:43:54.671 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00810902 s, 1.3 GB/s 2023-07-20T17:43:54.671 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2023-07-20T17:43:54.767 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:54 smithi079 systemd[1]: Starting Ceph mgr.y for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:43:54.768 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:54 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:54.517+0000 7f169628c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T17:43:54.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:54 smithi049 ceph-mon[96530]: Deploying daemon mgr.y on smithi079 2023-07-20T17:43:54.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:54 smithi049 ceph-mon[96530]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:55.066 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:54 smithi079 podman[99479]: 2023-07-20T17:43:55.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:54 smithi079 ceph-mon[98036]: Deploying daemon mgr.y on smithi079 2023-07-20T17:43:55.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:54 smithi079 ceph-mon[98036]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:55.084 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:54 smithi097 ceph-mon[98398]: Deploying daemon mgr.y on smithi079 2023-07-20T17:43:55.085 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:54 smithi097 ceph-mon[98398]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:55.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi049:vg_nvme/lv_4 2023-07-20T17:43:55.432 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:55 smithi079 bash[99479]: 2b5ad6812abf819c85a17318efa722cc1576feca4713946373aeca2b13c947ad 2023-07-20T17:43:55.433 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:55 smithi079 ceph-mgr[99519]: -- 172.21.15.79:0/1363113457 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5555f9a3f1e0 con 0x5555f9a14c00 2023-07-20T17:43:55.433 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:55 smithi079 systemd[1]: Started Ceph mgr.y for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:43:55.433 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:55 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:55.260+0000 7f54884b8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T17:43:55.433 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:55 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:55.341+0000 7f54884b8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T17:43:55.516 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:55 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:55.084+0000 7f169628c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T17:43:55.516 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:55 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:55.253+0000 7f169628c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T17:43:55.712 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:55 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:55.432+0000 7f54884b8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T17:43:55.713 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:55 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:55.530+0000 7f54884b8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T17:43:55.774 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:43:56.044 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:55 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:55.712+0000 7f54884b8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T17:43:56.316 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:56.043+0000 7f54884b8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T17:43:56.316 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:56.134+0000 7f54884b8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T17:43:56.316 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:56.268+0000 7f54884b8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T17:43:56.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:56 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:56 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:56 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:56 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:56 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:56.336 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:56.054+0000 7f169628c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T17:43:56.336 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:56.144+0000 7f169628c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T17:43:56.336 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:56 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.336 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:56 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.337 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:56 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.337 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:56 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.337 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:56 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:56.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:56 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:56 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:56 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:56 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:56.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:56 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:56.636 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:56.335+0000 7f169628c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T17:43:56.636 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:56.635+0000 7f169628c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T17:43:56.816 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:56.369+0000 7f54884b8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T17:43:56.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.987+0000 7f6916a07700 1 -- 172.21.15.49:0/2342746573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6910100900 msgr2=0x7f6910100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:56.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.987+0000 7f6916a07700 1 --2- 172.21.15.49:0/2342746573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6910100900 0x7f6910100d60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f69040099c0 tx=0x7f6904009cd0 comp rx=0 tx=0).stop 2023-07-20T17:43:56.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.987+0000 7f6916a07700 1 -- 172.21.15.49:0/2342746573 shutdown_connections 2023-07-20T17:43:56.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.987+0000 7f6916a07700 1 --2- 172.21.15.49:0/2342746573 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69100ff700 0x7f69100ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:56.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.987+0000 7f6916a07700 1 --2- 172.21.15.49:0/2342746573 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69101012a0 0x7f6910107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:56.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.987+0000 7f6916a07700 1 --2- 172.21.15.49:0/2342746573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6910100900 0x7f6910100d60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:56.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.987+0000 7f6916a07700 1 -- 172.21.15.49:0/2342746573 >> 172.21.15.49:0/2342746573 conn(0x7f69100fac80 msgr2=0x7f69100fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:43:56.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.987+0000 7f6916a07700 1 -- 172.21.15.49:0/2342746573 shutdown_connections 2023-07-20T17:43:56.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.987+0000 7f6916a07700 1 -- 172.21.15.49:0/2342746573 wait complete. 2023-07-20T17:43:56.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.988+0000 7f6916a07700 1 Processor -- start 2023-07-20T17:43:56.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.988+0000 7f6916a07700 1 -- start start 2023-07-20T17:43:56.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.988+0000 7f6916a07700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69100ff700 0x7f6910068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:56.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.988+0000 7f6916a07700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6910100900 0x7f691019b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:56.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.988+0000 7f6916a07700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69101012a0 0x7f691019d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:56.996 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.988+0000 7f6916a07700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691019e010 con 0x7f69101012a0 2023-07-20T17:43:56.996 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.988+0000 7f6916a07700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691019e180 con 0x7f6910100900 2023-07-20T17:43:56.996 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.988+0000 7f6916a07700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691019e340 con 0x7f69100ff700 2023-07-20T17:43:56.996 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f6914fa4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69101012a0 0x7f691019d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:56.997 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f690f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6910100900 0x7f691019b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:56.997 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f690ffff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69100ff700 0x7f6910068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:56.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f690f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6910100900 0x7f691019b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.49:57396/0 (socket says 172.21.15.49:57396) 2023-07-20T17:43:56.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f690ffff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69100ff700 0x7f6910068db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.49:59462/0 (socket says 172.21.15.49:59462) 2023-07-20T17:43:56.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f690f7fe700 1 -- 172.21.15.49:0/1195980264 learned_addr learned my addr 172.21.15.49:0/1195980264 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:43:56.999 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f6914fa4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69101012a0 0x7f691019d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35576/0 (socket says 172.21.15.49:35576) 2023-07-20T17:43:56.999 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f6914fa4700 1 -- 172.21.15.49:0/1195980264 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69100ff700 msgr2=0x7f6910068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:56.999 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f6914fa4700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69100ff700 0x7f6910068db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:57.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f6914fa4700 1 -- 172.21.15.49:0/1195980264 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6910100900 msgr2=0x7f691019b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:43:57.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f6914fa4700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6910100900 0x7f691019b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:43:57.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f6914fa4700 1 -- 172.21.15.49:0/1195980264 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69040095f0 con 0x7f69101012a0 2023-07-20T17:43:57.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f6914fa4700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69101012a0 0x7f691019d870 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f690800efc0 tx=0x7f690800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:57.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f690d7fa700 1 -- 172.21.15.49:0/1195980264 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6908018070 con 0x7f69101012a0 2023-07-20T17:43:57.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f690d7fa700 1 -- 172.21.15.49:0/1195980264 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f690800f040 con 0x7f69101012a0 2023-07-20T17:43:57.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f690d7fa700 1 -- 172.21.15.49:0/1195980264 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690800cd40 con 0x7f69101012a0 2023-07-20T17:43:57.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f6916a07700 1 -- 172.21.15.49:0/1195980264 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691019e5c0 con 0x7f69101012a0 2023-07-20T17:43:57.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.989+0000 7f6916a07700 1 -- 172.21.15.49:0/1195980264 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691019ea70 con 0x7f69101012a0 2023-07-20T17:43:57.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.990+0000 7f6916a07700 1 -- 172.21.15.49:0/1195980264 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6910106d50 con 0x7f69101012a0 2023-07-20T17:43:57.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.991+0000 7f690d7fa700 1 -- 172.21.15.49:0/1195980264 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f6908003a00 con 0x7f69101012a0 2023-07-20T17:43:57.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.991+0000 7f690d7fa700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68f80389e0 0x7f68f803aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:43:57.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.991+0000 7f690d7fa700 1 -- 172.21.15.49:0/1195980264 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f690804b860 con 0x7f69101012a0 2023-07-20T17:43:57.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.992+0000 7f690ffff700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68f80389e0 0x7f68f803aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:43:57.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.992+0000 7f690ffff700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68f80389e0 0x7f68f803aea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6910100760 tx=0x7f6900009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:43:57.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:56.996+0000 7f690d7fa700 1 -- 172.21.15.49:0/1195980264 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6908004ad0 con 0x7f69101012a0 2023-07-20T17:43:57.016 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:56.871+0000 7f169628c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T17:43:57.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:43:57.201+0000 7f6916a07700 1 -- 172.21.15.49:0/1195980264 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6910060bb0 con 0x7f68f80389e0 2023-07-20T17:43:57.214 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:56.827+0000 7f54884b8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T17:43:57.285 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:57.023+0000 7f169628c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T17:43:57.286 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-mon[98398]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:57.286 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.286 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.286 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.286 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.286 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:57.287 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:57.287 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.287 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T17:43:57.287 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T17:43:57.287 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:57.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:57 smithi049 ceph-mon[96530]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:57.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:57 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:57 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:57 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:57 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:57 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:57.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:57 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:57.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:57 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:57 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T17:43:57.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:57 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T17:43:57.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:57 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:57.566 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:57.405+0000 7f54884b8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T17:43:57.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-mon[98036]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:57.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:57.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:57.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:57.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T17:43:57.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T17:43:57.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:57.766 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:57.285+0000 7f169628c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T17:43:58.065 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:57.571+0000 7f54884b8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T17:43:58.266 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:57 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:57.975+0000 7f169628c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T17:43:58.266 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:58 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:58.103+0000 7f169628c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T17:43:58.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:58 smithi097 ceph-mon[98398]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T17:43:58.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:58 smithi097 ceph-mon[98398]: Reconfiguring daemon mgr.x on smithi049 2023-07-20T17:43:58.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:58 smithi097 ceph-mon[98398]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:58.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:43:58.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:43:58.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:58.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:58.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:58 smithi049 ceph-mon[96530]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T17:43:58.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:58 smithi049 ceph-mon[96530]: Reconfiguring daemon mgr.x on smithi049 2023-07-20T17:43:58.290 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:58 smithi049 ceph-mon[96530]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:58.290 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:43:58.290 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:43:58.290 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:58.290 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:58.566 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:58.329+0000 7f54884b8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T17:43:58.566 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:58.423+0000 7f54884b8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T17:43:58.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:58 smithi079 ceph-mon[98036]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T17:43:58.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:58 smithi079 ceph-mon[98036]: Reconfiguring daemon mgr.x on smithi049 2023-07-20T17:43:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:58 smithi079 ceph-mon[98036]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:43:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:43:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:43:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:58.623 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:58 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:58.293+0000 7f169628c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T17:43:58.624 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:58 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:58.404+0000 7f169628c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T17:43:58.864 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:58.592+0000 7f54884b8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T17:43:59.230 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:58.863+0000 7f54884b8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T17:43:59.230 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:59.086+0000 7f54884b8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T17:43:59.488 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:59.230+0000 7f54884b8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T17:43:59.488 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:59 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:59.489 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:59 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:59.489 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:59 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:59.489 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:59 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:59.489 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:59 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:59.489 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:59 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:59.489 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:43:59 smithi079 ceph-mon[98036]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:59.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:59 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:59.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:59 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:59.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:59 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:59.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:59 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:59.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:59 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:59.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:59 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:59.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:43:59 smithi049 ceph-mon[96530]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:59.766 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:58 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:58.623+0000 7f169628c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T17:43:59.766 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:43:59 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z[100531]: 2023-07-20T17:43:59.470+0000 7f169628c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T17:43:59.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:59 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:59.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:59 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:59.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:59 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:43:59.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:59 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:43:59.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:59 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:43:59.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:59 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:43:59.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:43:59 smithi097 ceph-mon[98398]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:43:59.816 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:43:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:43:59.487+0000 7f54884b8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T17:44:00.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:00.319+0000 7f690d7fa700 1 -- 172.21.15.49:0/1195980264 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7f690804a5f0 con 0x7f69101012a0 2023-07-20T17:44:00.491 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:44:00.175+0000 7f54884b8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T17:44:00.491 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:44:00.302+0000 7f54884b8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T17:44:00.492 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-mon[98036]: Standby manager daemon z started 2023-07-20T17:44:00.492 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-mon[98036]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T17:44:00.492 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-mon[98036]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T17:44:00.492 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-mon[98036]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T17:44:00.492 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-mon[98036]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T17:44:00.492 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3276502267' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63bfce41-a0a8-49b6-aa1e-ed7809c0627e"}]: dispatch 2023-07-20T17:44:00.493 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3276502267' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63bfce41-a0a8-49b6-aa1e-ed7809c0627e"}]': finished 2023-07-20T17:44:00.493 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-mon[98036]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T17:44:00.493 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:00.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:00 smithi049 ceph-mon[96530]: Standby manager daemon z started 2023-07-20T17:44:00.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:00 smithi049 ceph-mon[96530]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T17:44:00.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:00 smithi049 ceph-mon[96530]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T17:44:00.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:00 smithi049 ceph-mon[96530]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T17:44:00.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:00 smithi049 ceph-mon[96530]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T17:44:00.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:00 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3276502267' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63bfce41-a0a8-49b6-aa1e-ed7809c0627e"}]: dispatch 2023-07-20T17:44:00.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:00 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3276502267' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63bfce41-a0a8-49b6-aa1e-ed7809c0627e"}]': finished 2023-07-20T17:44:00.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:00 smithi049 ceph-mon[96530]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T17:44:00.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:00.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:00 smithi097 ceph-mon[98398]: Standby manager daemon z started 2023-07-20T17:44:00.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:00 smithi097 ceph-mon[98398]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T17:44:00.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:00 smithi097 ceph-mon[98398]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T17:44:00.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:00 smithi097 ceph-mon[98398]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T17:44:00.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:00 smithi097 ceph-mon[98398]: from='mgr.? 172.21.15.97:0/1210275686' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T17:44:00.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:00 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3276502267' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63bfce41-a0a8-49b6-aa1e-ed7809c0627e"}]: dispatch 2023-07-20T17:44:00.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:00 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3276502267' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63bfce41-a0a8-49b6-aa1e-ed7809c0627e"}]': finished 2023-07-20T17:44:00.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:00 smithi097 ceph-mon[98398]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T17:44:00.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:00.815 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:44:00.491+0000 7f54884b8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T17:44:00.815 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:44:00.602+0000 7f54884b8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T17:44:01.066 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:44:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:44:00.814+0000 7f54884b8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T17:44:01.662 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:01 smithi079 ceph-mon[98036]: mgrmap e14: x(active, since 67s), standbys: z 2023-07-20T17:44:01.662 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:01 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T17:44:01.662 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:01 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/138811108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:44:01.663 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:01 smithi079 ceph-mon[98036]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:01.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:01 smithi049 ceph-mon[96530]: mgrmap e14: x(active, since 67s), standbys: z 2023-07-20T17:44:01.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:01 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T17:44:01.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:01 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/138811108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:44:01.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:01 smithi049 ceph-mon[96530]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:01.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:01 smithi097 ceph-mon[98398]: mgrmap e14: x(active, since 67s), standbys: z 2023-07-20T17:44:01.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:01 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T17:44:01.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:01 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/138811108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:44:01.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:01 smithi097 ceph-mon[98398]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:02.066 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:44:01 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y[99495]: 2023-07-20T17:44:01.662+0000 7f54884b8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T17:44:02.331 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:02.330+0000 7f690d7fa700 1 -- 172.21.15.49:0/1195980264 <== mon.0 v2:172.21.15.49:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f6908004000 con 0x7f69101012a0 2023-07-20T17:44:02.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:02 smithi049 ceph-mon[96530]: Standby manager daemon y started 2023-07-20T17:44:02.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:02 smithi049 ceph-mon[96530]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T17:44:02.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:02 smithi049 ceph-mon[96530]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T17:44:02.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:02 smithi049 ceph-mon[96530]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T17:44:02.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:02 smithi049 ceph-mon[96530]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T17:44:02.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:02 smithi097 ceph-mon[98398]: Standby manager daemon y started 2023-07-20T17:44:02.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:02 smithi097 ceph-mon[98398]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T17:44:02.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:02 smithi097 ceph-mon[98398]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T17:44:02.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:02 smithi097 ceph-mon[98398]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T17:44:02.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:02 smithi097 ceph-mon[98398]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T17:44:02.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:02 smithi079 ceph-mon[98036]: Standby manager daemon y started 2023-07-20T17:44:02.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:02 smithi079 ceph-mon[98036]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T17:44:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:02 smithi079 ceph-mon[98036]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T17:44:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:02 smithi079 ceph-mon[98036]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T17:44:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:02 smithi079 ceph-mon[98036]: from='mgr.? 172.21.15.79:0/2994283318' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T17:44:03.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:03 smithi049 ceph-mon[96530]: mgrmap e15: x(active, since 69s), standbys: z, y 2023-07-20T17:44:03.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T17:44:03.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:03 smithi049 ceph-mon[96530]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:03.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:03 smithi097 ceph-mon[98398]: mgrmap e15: x(active, since 69s), standbys: z, y 2023-07-20T17:44:03.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T17:44:03.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:03 smithi097 ceph-mon[98398]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:03.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:03 smithi079 ceph-mon[98036]: mgrmap e15: x(active, since 69s), standbys: z, y 2023-07-20T17:44:03.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T17:44:03.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:03 smithi079 ceph-mon[98036]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:05.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:04 smithi049 ceph-mon[96530]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:05.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:04 smithi097 ceph-mon[98398]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:05.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:04 smithi079 ceph-mon[98036]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:06.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T17:44:06.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:07.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T17:44:07.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:07.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T17:44:07.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:07.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:07 smithi049 ceph-mon[96530]: Deploying daemon osd.0 on smithi049 2023-07-20T17:44:07.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:07 smithi049 ceph-mon[96530]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:08.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:07 smithi097 ceph-mon[98398]: Deploying daemon osd.0 on smithi049 2023-07-20T17:44:08.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:07 smithi097 ceph-mon[98398]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:08.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:07 smithi079 ceph-mon[98036]: Deploying daemon osd.0 on smithi049 2023-07-20T17:44:08.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:07 smithi079 ceph-mon[98036]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:08.949 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:08 smithi049 ceph-mon[96530]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:09.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:08 smithi097 ceph-mon[98398]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:09.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:08 smithi079 ceph-mon[98036]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:11.097 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:10 smithi049 ceph-mon[96530]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:11.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:10 smithi097 ceph-mon[98398]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:11.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:10 smithi079 ceph-mon[98036]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:12.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:12 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:12.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:12 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:12.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:12 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:12.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:12 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:12.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:12 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:12.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:12 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:12.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:12 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:12.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:12 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:12.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:12 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:13.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:13 smithi049 ceph-mon[96530]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:13.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:13 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:13.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:13 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:13.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:13 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:13.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:13 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:13.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:13 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:13.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:13 smithi097 ceph-mon[98398]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:13.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:13 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:13.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:13 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:13.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:13 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:13.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:13 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:13.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:13 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:13.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:13 smithi079 ceph-mon[98036]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:13.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:13 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:13.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:13 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:13.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:13 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:13.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:13 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:13.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:13 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:14.306 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.305+0000 7f690d7fa700 1 -- 172.21.15.49:0/1195980264 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6910060bb0 con 0x7f68f80389e0 2023-07-20T17:44:14.307 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 0 on host 'smithi049' 2023-07-20T17:44:14.310 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.309+0000 7f6916a07700 1 -- 172.21.15.49:0/1195980264 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68f80389e0 msgr2=0x7f68f803aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:14.311 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.309+0000 7f6916a07700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68f80389e0 0x7f68f803aea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6910100760 tx=0x7f6900009260 comp rx=0 tx=0).stop 2023-07-20T17:44:14.311 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.309+0000 7f6916a07700 1 -- 172.21.15.49:0/1195980264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69101012a0 msgr2=0x7f691019d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:14.311 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.309+0000 7f6916a07700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69101012a0 0x7f691019d870 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f690800efc0 tx=0x7f690800c3a0 comp rx=0 tx=0).stop 2023-07-20T17:44:14.311 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.309+0000 7f6916a07700 1 -- 172.21.15.49:0/1195980264 shutdown_connections 2023-07-20T17:44:14.311 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.309+0000 7f6916a07700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f68f80389e0 0x7f68f803aea0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:14.311 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.309+0000 7f6916a07700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69100ff700 0x7f6910068db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:14.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.309+0000 7f6916a07700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69101012a0 0x7f691019d870 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:14.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.309+0000 7f6916a07700 1 --2- 172.21.15.49:0/1195980264 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6910100900 0x7f691019b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:14.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.309+0000 7f6916a07700 1 -- 172.21.15.49:0/1195980264 >> 172.21.15.49:0/1195980264 conn(0x7f69100fac80 msgr2=0x7f6910103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:44:14.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.310+0000 7f6916a07700 1 -- 172.21.15.49:0/1195980264 shutdown_connections 2023-07-20T17:44:14.313 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:44:14.310+0000 7f6916a07700 1 -- 172.21.15.49:0/1195980264 wait complete. 2023-07-20T17:44:14.860 DEBUG:teuthology.orchestra.run.smithi049:osd.0> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.0.service 2023-07-20T17:44:14.862 INFO:tasks.cephadm:Deploying osd.1 on smithi079 with /dev/vg_nvme/lv_4... 2023-07-20T17:44:14.863 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T17:44:14.897 INFO:journalctl@ceph.osd.0.smithi049.stdout:-- Logs begin at Thu 2023-07-20 17:16:49 UTC. -- 2023-07-20T17:44:15.409 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T17:44:15.425 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:44:15 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-0[109572]: 2023-07-20T17:44:15.129+0000 7faaacfce640 -1 osd.0 0 log_to_monitors true 2023-07-20T17:44:15.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:15 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:15.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:15 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:15.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:15 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:15.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:15 smithi049 ceph-mon[96530]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:15.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:15 smithi049 ceph-mon[96530]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T17:44:15.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:15 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:15.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:15 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:15.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:15 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:15.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:15 smithi079 ceph-mon[98036]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:15.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:15 smithi079 ceph-mon[98036]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T17:44:15.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:15 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:15.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:15 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:15.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:15 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:15.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:15 smithi097 ceph-mon[98398]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:15.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:15 smithi097 ceph-mon[98398]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T17:44:16.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:16 smithi049 ceph-mon[96530]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T17:44:16.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:16 smithi049 ceph-mon[96530]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T17:44:16.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:16 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:16.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:16 smithi049 ceph-mon[96530]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:44:16.490 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:44:16 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-0[109572]: 2023-07-20T17:44:16.303+0000 7faaa2646700 -1 osd.0 0 waiting for initial osdmap 2023-07-20T17:44:16.490 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:44:16 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-0[109572]: 2023-07-20T17:44:16.307+0000 7faa9bc37700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:44:16.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:16 smithi079 ceph-mon[98036]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T17:44:16.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:16 smithi079 ceph-mon[98036]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T17:44:16.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:16 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:16.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:16 smithi079 ceph-mon[98036]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:44:16.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:16 smithi097 ceph-mon[98398]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T17:44:16.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:16 smithi097 ceph-mon[98398]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T17:44:16.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:16 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:16.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:16 smithi097 ceph-mon[98398]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:44:16.915 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T17:44:16.915 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T17:44:16.915 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2023-07-20T17:44:16.915 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2023-07-20T17:44:16.915 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110005 s, 953 MB/s 2023-07-20T17:44:16.916 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2023-07-20T17:44:17.261 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi079:vg_nvme/lv_4 2023-07-20T17:44:17.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:44:17.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T17:44:17.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:17.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:17.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:17.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:44:17.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:17.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:17.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:17 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:44:17.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T17:44:17.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:17.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:17.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:17.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:44:17.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:17.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:17.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:17 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: from='osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:44:17.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T17:44:17.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:17.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:17.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T17:44:17.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:44:17.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:17.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:17.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:17 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:17.809 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T17:44:18.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:18 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:44:18.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:18 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:44:18.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:18 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi049 2023-07-20T17:44:18.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:18 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi049 to 17168M 2023-07-20T17:44:18.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:18.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:18 smithi079 ceph-mon[98036]: osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776] boot 2023-07-20T17:44:18.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:18 smithi079 ceph-mon[98036]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T17:44:18.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:18.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:18 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:44:18.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:18 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:44:18.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:18 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi049 2023-07-20T17:44:18.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:18 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi049 to 17168M 2023-07-20T17:44:18.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:18.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:18 smithi049 ceph-mon[96530]: osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776] boot 2023-07-20T17:44:18.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:18 smithi049 ceph-mon[96530]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T17:44:18.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:18.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:18 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:44:18.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:18 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:44:18.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:18 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi049 2023-07-20T17:44:18.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:18 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi049 to 17168M 2023-07-20T17:44:18.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:18.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:18 smithi097 ceph-mon[98398]: osd.0 [v2:172.21.15.49:6802/2306317776,v1:172.21.15.49:6803/2306317776] boot 2023-07-20T17:44:18.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:18 smithi097 ceph-mon[98398]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T17:44:18.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T17:44:18.978 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.977+0000 7fce21476700 1 -- 172.21.15.79:0/2465832358 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c100a30 msgr2=0x7fce1c107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:18.978 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.977+0000 7fce21476700 1 --2- 172.21.15.79:0/2465832358 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c100a30 0x7fce1c107350 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fce1400b0a0 tx=0x7fce1400b3b0 comp rx=0 tx=0).stop 2023-07-20T17:44:18.979 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.977+0000 7fce21476700 1 -- 172.21.15.79:0/2465832358 shutdown_connections 2023-07-20T17:44:18.979 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.977+0000 7fce21476700 1 --2- 172.21.15.79:0/2465832358 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c100a30 0x7fce1c107350 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:18.979 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.977+0000 7fce21476700 1 --2- 172.21.15.79:0/2465832358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce1c0fee80 0x7fce1c0ff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:18.979 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.977+0000 7fce21476700 1 --2- 172.21.15.79:0/2465832358 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce1c100070 0x7fce1c1004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:18.979 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.977+0000 7fce21476700 1 -- 172.21.15.79:0/2465832358 >> 172.21.15.79:0/2465832358 conn(0x7fce1c0fa450 msgr2=0x7fce1c0fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:44:18.979 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.977+0000 7fce21476700 1 -- 172.21.15.79:0/2465832358 shutdown_connections 2023-07-20T17:44:18.979 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.977+0000 7fce21476700 1 -- 172.21.15.79:0/2465832358 wait complete. 2023-07-20T17:44:18.981 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.977+0000 7fce21476700 1 Processor -- start 2023-07-20T17:44:18.981 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce21476700 1 -- start start 2023-07-20T17:44:18.981 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce21476700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c0fee80 0x7fce1c103720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:44:18.982 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce21476700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce1c100070 0x7fce1c101d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:44:18.982 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce21476700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce1c100a30 0x7fce1c1022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:44:18.982 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce21476700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c102a30 con 0x7fce1c100a30 2023-07-20T17:44:18.982 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce21476700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c102ba0 con 0x7fce1c0fee80 2023-07-20T17:44:18.982 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce21476700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c1a0670 con 0x7fce1c100070 2023-07-20T17:44:18.982 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c0fee80 0x7fce1c103720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:44:18.983 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c0fee80 0x7fce1c103720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47746/0 (socket says 172.21.15.79:47746) 2023-07-20T17:44:18.983 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1affd700 1 -- 172.21.15.79:0/2696903101 learned_addr learned my addr 172.21.15.79:0/2696903101 (peer_addr_for_me v2:172.21.15.79:0/0) 2023-07-20T17:44:18.983 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1a7fc700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce1c100070 0x7fce1c101d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:44:18.983 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1b7fe700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce1c100a30 0x7fce1c1022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:44:18.983 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1affd700 1 -- 172.21.15.79:0/2696903101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce1c100070 msgr2=0x7fce1c101d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:18.984 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1affd700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce1c100070 0x7fce1c101d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:18.984 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1affd700 1 -- 172.21.15.79:0/2696903101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce1c100a30 msgr2=0x7fce1c1022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:18.984 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1affd700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce1c100a30 0x7fce1c1022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:18.985 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1affd700 1 -- 172.21.15.79:0/2696903101 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce14009d00 con 0x7fce1c0fee80 2023-07-20T17:44:18.985 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1a7fc700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce1c100070 0x7fce1c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:44:18.985 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1b7fe700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce1c100a30 0x7fce1c1022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:44:18.985 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce1affd700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c0fee80 0x7fce1c103720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00c970 tx=0x7fce0c00cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:44:18.986 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce03fff700 1 -- 172.21.15.79:0/2696903101 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c004220 con 0x7fce1c0fee80 2023-07-20T17:44:18.986 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce03fff700 1 -- 172.21.15.79:0/2696903101 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce0c007d60 con 0x7fce1c0fee80 2023-07-20T17:44:18.986 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce03fff700 1 -- 172.21.15.79:0/2696903101 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c003d00 con 0x7fce1c0fee80 2023-07-20T17:44:18.986 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.978+0000 7fce21476700 1 -- 172.21.15.79:0/2696903101 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1c1a0950 con 0x7fce1c0fee80 2023-07-20T17:44:18.986 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.979+0000 7fce21476700 1 -- 172.21.15.79:0/2696903101 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1c1a0e00 con 0x7fce1c0fee80 2023-07-20T17:44:18.986 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.979+0000 7fce21476700 1 -- 172.21.15.79:0/2696903101 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1c10a700 con 0x7fce1c0fee80 2023-07-20T17:44:18.987 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.980+0000 7fce03fff700 1 -- 172.21.15.79:0/2696903101 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fce0c004380 con 0x7fce1c0fee80 2023-07-20T17:44:18.987 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.981+0000 7fce03fff700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fce040a10c0 0x7fce040a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:44:18.988 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.981+0000 7fce03fff700 1 -- 172.21.15.79:0/2696903101 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fce0c0cae50 con 0x7fce1c0fee80 2023-07-20T17:44:18.988 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.981+0000 7fce1a7fc700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fce040a10c0 0x7fce040a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:44:18.988 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.982+0000 7fce1a7fc700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fce040a10c0 0x7fce040a3580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fce1c0febb0 tx=0x7fce10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:44:18.988 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:18.985+0000 7fce03fff700 1 -- 172.21.15.79:0/2696903101 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fce0c096010 con 0x7fce1c0fee80 2023-07-20T17:44:19.180 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:19.179+0000 7fce21476700 1 -- 172.21.15.79:0/2696903101 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fce1c0fc3b0 con 0x7fce040a10c0 2023-07-20T17:44:19.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:19 smithi049 ceph-mon[96530]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T17:44:19.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:19 smithi049 ceph-mon[96530]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:19.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:44:19.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:44:19.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:19.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:19 smithi097 ceph-mon[98398]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T17:44:19.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:19 smithi097 ceph-mon[98398]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:19.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:19 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:44:19.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:19 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:44:19.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:19 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:19.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:19 smithi079 ceph-mon[98036]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T17:44:19.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:19 smithi079 ceph-mon[98036]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:19.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:19 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:44:19.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:19 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:44:19.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:19 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:20.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:20 smithi079 ceph-mon[98036]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:44:20.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:20 smithi049 ceph-mon[96530]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:44:20.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:20 smithi097 ceph-mon[98398]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:44:21.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:21 smithi049 ceph-mon[96530]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:21.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:21 smithi097 ceph-mon[98398]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:21.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:21 smithi079 ceph-mon[98036]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:22.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:22 smithi049 ceph-mon[96530]: from='client.? 172.21.15.79:0/4193532495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48cd9d65-b37d-410a-8007-f53fbd24dd73"}]: dispatch 2023-07-20T17:44:22.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:22 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48cd9d65-b37d-410a-8007-f53fbd24dd73"}]: dispatch 2023-07-20T17:44:22.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:22 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48cd9d65-b37d-410a-8007-f53fbd24dd73"}]': finished 2023-07-20T17:44:22.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:22 smithi049 ceph-mon[96530]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T17:44:22.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:22 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:22.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:22 smithi049 ceph-mon[96530]: from='client.? 172.21.15.79:0/1212223649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:44:22.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:22 smithi097 ceph-mon[98398]: from='client.? 172.21.15.79:0/4193532495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48cd9d65-b37d-410a-8007-f53fbd24dd73"}]: dispatch 2023-07-20T17:44:22.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:22 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48cd9d65-b37d-410a-8007-f53fbd24dd73"}]: dispatch 2023-07-20T17:44:22.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:22 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48cd9d65-b37d-410a-8007-f53fbd24dd73"}]': finished 2023-07-20T17:44:22.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:22 smithi097 ceph-mon[98398]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T17:44:22.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:22 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:22.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:22 smithi097 ceph-mon[98398]: from='client.? 172.21.15.79:0/1212223649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:44:22.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:22 smithi079 ceph-mon[98036]: from='client.? 172.21.15.79:0/4193532495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48cd9d65-b37d-410a-8007-f53fbd24dd73"}]: dispatch 2023-07-20T17:44:22.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:22 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48cd9d65-b37d-410a-8007-f53fbd24dd73"}]: dispatch 2023-07-20T17:44:22.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:22 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48cd9d65-b37d-410a-8007-f53fbd24dd73"}]': finished 2023-07-20T17:44:22.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:22 smithi079 ceph-mon[98036]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T17:44:22.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:22 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:22.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:22 smithi079 ceph-mon[98036]: from='client.? 172.21.15.79:0/1212223649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:44:23.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:23 smithi049 ceph-mon[96530]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:23.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:23 smithi097 ceph-mon[98398]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:23.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:23 smithi079 ceph-mon[98036]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:25.204 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:24 smithi079 ceph-mon[98036]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:25.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:24 smithi049 ceph-mon[96530]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:25.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:24 smithi097 ceph-mon[98398]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:27.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:26 smithi079 ceph-mon[98036]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:27.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:26 smithi049 ceph-mon[96530]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:27.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:26 smithi097 ceph-mon[98398]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:28.682 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T17:44:28.682 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:28.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T17:44:28.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:28.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T17:44:28.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:29.624 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:29 smithi079 ceph-mon[98036]: Deploying daemon osd.1 on smithi079 2023-07-20T17:44:29.625 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:29 smithi079 ceph-mon[98036]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:29.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:29 smithi049 ceph-mon[96530]: Deploying daemon osd.1 on smithi079 2023-07-20T17:44:29.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:29 smithi049 ceph-mon[96530]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:29.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:29 smithi097 ceph-mon[98398]: Deploying daemon osd.1 on smithi079 2023-07-20T17:44:29.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:29 smithi097 ceph-mon[98398]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:32.227 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:31 smithi079 ceph-mon[98036]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:32.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:31 smithi049 ceph-mon[96530]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:32.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:31 smithi097 ceph-mon[98398]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:33.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:33 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:33.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:33 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:33.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:33 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:33.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:33 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:33.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:33 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:33.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:33 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:34.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:33 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:34.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:33 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:34.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:33 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:34.746 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:34 smithi079 ceph-mon[98036]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:34.747 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:34.747 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:34.747 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:34.748 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:34.748 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:34.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:34 smithi049 ceph-mon[96530]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:34.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:34.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:34.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:34.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:34.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:35.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:34 smithi097 ceph-mon[98398]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:35.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:35.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:35.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:35.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:35.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:35.044 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 1 on host 'smithi079' 2023-07-20T17:44:35.045 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.043+0000 7fce03fff700 1 -- 172.21.15.79:0/2696903101 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fce1c0fc3b0 con 0x7fce040a10c0 2023-07-20T17:44:35.046 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 -- 172.21.15.79:0/2696903101 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fce040a10c0 msgr2=0x7fce040a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:35.046 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fce040a10c0 0x7fce040a3580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fce1c0febb0 tx=0x7fce10005f90 comp rx=0 tx=0).stop 2023-07-20T17:44:35.046 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 -- 172.21.15.79:0/2696903101 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c0fee80 msgr2=0x7fce1c103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c0fee80 0x7fce1c103720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00c970 tx=0x7fce0c00cd30 comp rx=0 tx=0).stop 2023-07-20T17:44:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 -- 172.21.15.79:0/2696903101 shutdown_connections 2023-07-20T17:44:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fce040a10c0 0x7fce040a3580 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce1c100a30 0x7fce1c1022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce1c0fee80 0x7fce1c103720 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 --2- 172.21.15.79:0/2696903101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce1c100070 0x7fce1c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 -- 172.21.15.79:0/2696903101 >> 172.21.15.79:0/2696903101 conn(0x7fce1c0fa450 msgr2=0x7fce1c0fbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 -- 172.21.15.79:0/2696903101 shutdown_connections 2023-07-20T17:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:44:35.046+0000 7fce21476700 1 -- 172.21.15.79:0/2696903101 wait complete. 2023-07-20T17:44:35.496 DEBUG:teuthology.orchestra.run.smithi079:osd.1> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.1.service 2023-07-20T17:44:35.498 INFO:tasks.cephadm:Deploying osd.2 on smithi097 with /dev/vg_nvme/lv_4... 2023-07-20T17:44:35.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T17:44:35.527 INFO:journalctl@ceph.osd.1.smithi079.stdout:-- Logs begin at Thu 2023-07-20 17:15:42 UTC. -- 2023-07-20T17:44:36.055 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:44:36.159 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:36 smithi079 ceph-mon[98036]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:36.159 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:36.160 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:36.160 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:36.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:36 smithi097 ceph-mon[98398]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:36.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:36.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:36.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:36.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:36 smithi049 ceph-mon[96530]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:36.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:36.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:36.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:36.816 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:44:36 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-1[103103]: 2023-07-20T17:44:36.499+0000 7f0a78a0d640 -1 osd.1 0 log_to_monitors true 2023-07-20T17:44:37.171 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:37 smithi097 ceph-mon[98398]: from='osd.1 [v2:172.21.15.79:6800/2048064928,v1:172.21.15.79:6801/2048064928]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T17:44:37.171 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:37 smithi097 ceph-mon[98398]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T17:44:37.309 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:37 smithi079 ceph-mon[98036]: from='osd.1 [v2:172.21.15.79:6800/2048064928,v1:172.21.15.79:6801/2048064928]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T17:44:37.310 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:37 smithi079 ceph-mon[98036]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T17:44:37.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:37 smithi049 ceph-mon[96530]: from='osd.1 [v2:172.21.15.79:6800/2048064928,v1:172.21.15.79:6801/2048064928]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T17:44:37.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:37 smithi049 ceph-mon[96530]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T17:44:37.632 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T17:44:37.632 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T17:44:37.632 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2023-07-20T17:44:37.633 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2023-07-20T17:44:37.633 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134865 s, 777 MB/s 2023-07-20T17:44:37.633 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2023-07-20T17:44:38.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi097:vg_nvme/lv_4 2023-07-20T17:44:38.316 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-1[103103]: 2023-07-20T17:44:38.022+0000 7f0a6e085700 -1 osd.1 0 waiting for initial osdmap 2023-07-20T17:44:38.317 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-1[103103]: 2023-07-20T17:44:38.025+0000 7f0a65e73700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:44:38.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:38.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T17:44:38.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T17:44:38.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:38.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: from='osd.1 [v2:172.21.15.79:6800/2048064928,v1:172.21.15.79:6801/2048064928]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:44:38.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:44:38.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi079 2023-07-20T17:44:38.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:44:38.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi079 to 17168M 2023-07-20T17:44:38.320 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.320 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:38.320 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:38.320 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:38.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T17:44:38.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T17:44:38.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:38.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: from='osd.1 [v2:172.21.15.79:6800/2048064928,v1:172.21.15.79:6801/2048064928]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:44:38.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:44:38.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi079 2023-07-20T17:44:38.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:44:38.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi079 to 17168M 2023-07-20T17:44:38.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:38.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:38.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:38.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T17:44:38.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T17:44:38.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:38.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: from='osd.1 [v2:172.21.15.79:6800/2048064928,v1:172.21.15.79:6801/2048064928]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:44:38.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:44:38.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi079 2023-07-20T17:44:38.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:44:38.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi079 to 17168M 2023-07-20T17:44:38.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:38.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:38.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:38.645 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:44:39.243 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:39 smithi097 ceph-mon[98398]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:44:39.243 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:39 smithi097 ceph-mon[98398]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T17:44:39.243 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:39 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:39.243 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:39 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:39.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:39 smithi079 ceph-mon[98036]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:44:39.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:39 smithi079 ceph-mon[98036]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T17:44:39.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:39 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:39.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:39 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:39.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:39 smithi049 ceph-mon[96530]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:44:39.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:39 smithi049 ceph-mon[96530]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T17:44:39.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:39 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:39.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:39 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:39.991 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.988+0000 7fee2ecf2700 1 -- 172.21.15.97:0/2611588029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee280ff700 msgr2=0x7fee280ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:39.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.988+0000 7fee2ecf2700 1 --2- 172.21.15.97:0/2611588029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee280ff700 0x7fee280ffb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009990 tx=0x7fee1c009ca0 comp rx=0 tx=0).stop 2023-07-20T17:44:39.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.988+0000 7fee2ecf2700 1 -- 172.21.15.97:0/2611588029 shutdown_connections 2023-07-20T17:44:39.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.988+0000 7fee2ecf2700 1 --2- 172.21.15.97:0/2611588029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee280ff700 0x7fee280ffb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:39.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.988+0000 7fee2ecf2700 1 --2- 172.21.15.97:0/2611588029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee281012a0 0x7fee28107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:39.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.988+0000 7fee2ecf2700 1 --2- 172.21.15.97:0/2611588029 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee28100900 0x7fee28100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:39.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.988+0000 7fee2ecf2700 1 -- 172.21.15.97:0/2611588029 >> 172.21.15.97:0/2611588029 conn(0x7fee280fac80 msgr2=0x7fee280fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:44:39.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.988+0000 7fee2ecf2700 1 -- 172.21.15.97:0/2611588029 shutdown_connections 2023-07-20T17:44:39.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.988+0000 7fee2ecf2700 1 -- 172.21.15.97:0/2611588029 wait complete. 2023-07-20T17:44:39.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee2ecf2700 1 Processor -- start 2023-07-20T17:44:39.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee2ecf2700 1 -- start start 2023-07-20T17:44:39.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee2ecf2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee280ff700 0x7fee28195430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:44:39.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee2ecf2700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee28100900 0x7fee28195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:44:39.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee2ecf2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee281012a0 0x7fee2819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:44:39.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee2ecf2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2819cfc0 con 0x7fee281012a0 2023-07-20T17:44:39.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee2ecf2700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2819d100 con 0x7fee280ff700 2023-07-20T17:44:39.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee2ecf2700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2819d290 con 0x7fee28100900 2023-07-20T17:44:39.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee27fff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee28100900 0x7fee28195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:44:39.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee27fff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee28100900 0x7fee28195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60664/0 (socket says 172.21.15.97:60664) 2023-07-20T17:44:39.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee27fff700 1 -- 172.21.15.97:0/3172936120 learned_addr learned my addr 172.21.15.97:0/3172936120 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-07-20T17:44:39.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee27fff700 1 -- 172.21.15.97:0/3172936120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee280ff700 msgr2=0x7fee28195430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:39.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee27fff700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee280ff700 0x7fee28195430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:39.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee27fff700 1 -- 172.21.15.97:0/3172936120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee281012a0 msgr2=0x7fee2819c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:39.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee2d28f700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee281012a0 0x7fee2819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:44:39.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee27fff700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee281012a0 0x7fee2819c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:39.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee27fff700 1 -- 172.21.15.97:0/3172936120 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c0095f0 con 0x7fee28100900 2023-07-20T17:44:39.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.989+0000 7fee2d28f700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee281012a0 0x7fee2819c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T17:44:39.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.990+0000 7fee27fff700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee28100900 0x7fee28195970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fee1400b560 tx=0x7fee1400b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:44:39.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.990+0000 7fee25ffb700 1 -- 172.21.15.97:0/3172936120 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14011830 con 0x7fee28100900 2023-07-20T17:44:39.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.990+0000 7fee2ecf2700 1 -- 172.21.15.97:0/3172936120 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2819d570 con 0x7fee28100900 2023-07-20T17:44:39.996 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.990+0000 7fee25ffb700 1 -- 172.21.15.97:0/3172936120 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee14011e50 con 0x7fee28100900 2023-07-20T17:44:39.996 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.990+0000 7fee25ffb700 1 -- 172.21.15.97:0/3172936120 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1400f2e0 con 0x7fee28100900 2023-07-20T17:44:39.996 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.990+0000 7fee2ecf2700 1 -- 172.21.15.97:0/3172936120 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee28068480 con 0x7fee28100900 2023-07-20T17:44:39.997 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.991+0000 7fee2ecf2700 1 -- 172.21.15.97:0/3172936120 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2810af80 con 0x7fee28100900 2023-07-20T17:44:39.997 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.992+0000 7fee25ffb700 1 -- 172.21.15.97:0/3172936120 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fee14011990 con 0x7fee28100900 2023-07-20T17:44:39.997 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.992+0000 7fee25ffb700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee180a32d0 0x7fee180a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:44:39.997 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.992+0000 7fee25ffb700 1 -- 172.21.15.97:0/3172936120 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fee140cab60 con 0x7fee28100900 2023-07-20T17:44:39.998 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.992+0000 7fee2ca8e700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee180a32d0 0x7fee180a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:44:39.998 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.993+0000 7fee2ca8e700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee180a32d0 0x7fee180a5790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee1c005b40 tx=0x7fee1c005ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:44:39.999 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:39.997+0000 7fee25ffb700 1 -- 172.21.15.97:0/3172936120 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fee14095b00 con 0x7fee28100900 2023-07-20T17:44:40.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:40.194+0000 7fee2ecf2700 1 -- 172.21.15.97:0/3172936120 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fee280690d0 con 0x7fee180a32d0 2023-07-20T17:44:40.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:40 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:44:40.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:40 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:44:40.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:40 smithi097 ceph-mon[98398]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:40.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:40 smithi097 ceph-mon[98398]: osd.1 [v2:172.21.15.79:6800/2048064928,v1:172.21.15.79:6801/2048064928] boot 2023-07-20T17:44:40.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:40 smithi097 ceph-mon[98398]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T17:44:40.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:40 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:40.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:40 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:44:40.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:40 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:44:40.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:40 smithi079 ceph-mon[98036]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:40.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:40 smithi079 ceph-mon[98036]: osd.1 [v2:172.21.15.79:6800/2048064928,v1:172.21.15.79:6801/2048064928] boot 2023-07-20T17:44:40.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:40 smithi079 ceph-mon[98036]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T17:44:40.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:40 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:40.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:40 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:44:40.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:40 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:44:40.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:40 smithi049 ceph-mon[96530]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T17:44:40.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:40 smithi049 ceph-mon[96530]: osd.1 [v2:172.21.15.79:6800/2048064928,v1:172.21.15.79:6801/2048064928] boot 2023-07-20T17:44:40.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:40 smithi049 ceph-mon[96530]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T17:44:40.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:40 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T17:44:41.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:41 smithi097 ceph-mon[98398]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T17:44:41.266 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:41 smithi097 ceph-mon[98398]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:44:41.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:44:41.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:44:41.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:41.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:41 smithi079 ceph-mon[98036]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T17:44:41.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:41 smithi079 ceph-mon[98036]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:44:41.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:44:41.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:44:41.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:41.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:41 smithi049 ceph-mon[96530]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T17:44:41.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:41 smithi049 ceph-mon[96530]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:44:41.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:44:41.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:44:41.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:42.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:42 smithi097 ceph-mon[98398]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:42.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:42 smithi079 ceph-mon[98036]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:42.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:42 smithi049 ceph-mon[96530]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:43.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:43 smithi049 ceph-mon[96530]: from='client.? 172.21.15.97:0/2313850872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "272a17dd-e825-4990-a178-c2eb6985f37c"}]: dispatch 2023-07-20T17:44:43.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:43 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "272a17dd-e825-4990-a178-c2eb6985f37c"}]: dispatch 2023-07-20T17:44:43.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:43 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "272a17dd-e825-4990-a178-c2eb6985f37c"}]': finished 2023-07-20T17:44:43.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:43 smithi049 ceph-mon[96530]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T17:44:43.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:43 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:44:43.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:43 smithi049 ceph-mon[96530]: from='client.? 172.21.15.97:0/4254398429' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:44:43.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:43 smithi097 ceph-mon[98398]: from='client.? 172.21.15.97:0/2313850872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "272a17dd-e825-4990-a178-c2eb6985f37c"}]: dispatch 2023-07-20T17:44:43.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:43 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "272a17dd-e825-4990-a178-c2eb6985f37c"}]: dispatch 2023-07-20T17:44:43.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:43 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "272a17dd-e825-4990-a178-c2eb6985f37c"}]': finished 2023-07-20T17:44:43.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:43 smithi097 ceph-mon[98398]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T17:44:43.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:43 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:44:43.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:43 smithi097 ceph-mon[98398]: from='client.? 172.21.15.97:0/4254398429' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:44:43.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:43 smithi079 ceph-mon[98036]: from='client.? 172.21.15.97:0/2313850872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "272a17dd-e825-4990-a178-c2eb6985f37c"}]: dispatch 2023-07-20T17:44:43.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:43 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "272a17dd-e825-4990-a178-c2eb6985f37c"}]: dispatch 2023-07-20T17:44:43.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:43 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "272a17dd-e825-4990-a178-c2eb6985f37c"}]': finished 2023-07-20T17:44:43.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:43 smithi079 ceph-mon[98036]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T17:44:43.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:43 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:44:43.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:43 smithi079 ceph-mon[98036]: from='client.? 172.21.15.97:0/4254398429' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:44:44.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:44 smithi049 ceph-mon[96530]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:45.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:44 smithi097 ceph-mon[98398]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:45.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:44 smithi079 ceph-mon[98036]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:46.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:46 smithi049 ceph-mon[96530]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:47.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:46 smithi097 ceph-mon[98398]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:47.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:46 smithi079 ceph-mon[98036]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:48.899 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:48 smithi097 ceph-mon[98398]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:48.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:48 smithi049 ceph-mon[96530]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:49.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:48 smithi079 ceph-mon[98036]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:49.955 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:49 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T17:44:49.955 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:49 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:49.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:49 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T17:44:49.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:49 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:50.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:49 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T17:44:50.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:49 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:50.882 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:50 smithi097 ceph-mon[98398]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:50.882 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:50 smithi097 ceph-mon[98398]: Deploying daemon osd.2 on smithi097 2023-07-20T17:44:50.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:50 smithi049 ceph-mon[96530]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:50.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:50 smithi049 ceph-mon[96530]: Deploying daemon osd.2 on smithi097 2023-07-20T17:44:51.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:50 smithi079 ceph-mon[98036]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:51.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:50 smithi079 ceph-mon[98036]: Deploying daemon osd.2 on smithi097 2023-07-20T17:44:52.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:52 smithi097 ceph-mon[98398]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:52.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:52 smithi049 ceph-mon[96530]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:53.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:52 smithi079 ceph-mon[98036]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:54.925 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:54 smithi097 ceph-mon[98398]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:54.926 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:54 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:54.926 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:54 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:54.926 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:54 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:54.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:54 smithi049 ceph-mon[96530]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:54.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:54 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:54.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:54 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:54.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:54 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:55.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:54 smithi079 ceph-mon[98036]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:55.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:54 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:55.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:54 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:55.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:54 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:56.996 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:56 smithi049 ceph-mon[96530]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:56.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:56 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:56.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:56 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:56.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:56 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:56.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:56 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:56.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:56 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:57.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:56 smithi097 ceph-mon[98398]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:57.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:56 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:57.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:56 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:57.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:56 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:57.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:56 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:57.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:56 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:57.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:56 smithi079 ceph-mon[98036]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:57.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:56 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:57.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:56 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:57.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:56 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:44:57.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:56 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:44:57.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:56 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:57.160 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.159+0000 7fee25ffb700 1 -- 172.21.15.97:0/3172936120 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fee280690d0 con 0x7fee180a32d0 2023-07-20T17:44:57.161 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 2 on host 'smithi097' 2023-07-20T17:44:57.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.161+0000 7fee2ecf2700 1 -- 172.21.15.97:0/3172936120 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee180a32d0 msgr2=0x7fee180a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:57.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.162+0000 7fee2ecf2700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee180a32d0 0x7fee180a5790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee1c005b40 tx=0x7fee1c005ad0 comp rx=0 tx=0).stop 2023-07-20T17:44:57.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.162+0000 7fee2ecf2700 1 -- 172.21.15.97:0/3172936120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee28100900 msgr2=0x7fee28195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:44:57.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.162+0000 7fee2ecf2700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee28100900 0x7fee28195970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fee1400b560 tx=0x7fee1400b920 comp rx=0 tx=0).stop 2023-07-20T17:44:57.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.162+0000 7fee2ecf2700 1 -- 172.21.15.97:0/3172936120 shutdown_connections 2023-07-20T17:44:57.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.162+0000 7fee2ecf2700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fee180a32d0 0x7fee180a5790 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:57.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.162+0000 7fee2ecf2700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee281012a0 0x7fee2819c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:57.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.162+0000 7fee2ecf2700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee280ff700 0x7fee28195430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:57.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.162+0000 7fee2ecf2700 1 --2- 172.21.15.97:0/3172936120 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee28100900 0x7fee28195970 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:44:57.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.162+0000 7fee2ecf2700 1 -- 172.21.15.97:0/3172936120 >> 172.21.15.97:0/3172936120 conn(0x7fee280fac80 msgr2=0x7fee280fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:44:57.164 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.162+0000 7fee2ecf2700 1 -- 172.21.15.97:0/3172936120 shutdown_connections 2023-07-20T17:44:57.164 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:44:57.162+0000 7fee2ecf2700 1 -- 172.21.15.97:0/3172936120 wait complete. 2023-07-20T17:44:57.537 DEBUG:teuthology.orchestra.run.smithi097:osd.2> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.2.service 2023-07-20T17:44:57.539 INFO:tasks.cephadm:Deploying osd.3 on smithi049 with /dev/vg_nvme/lv_3... 2023-07-20T17:44:57.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T17:44:57.567 INFO:journalctl@ceph.osd.2.smithi097.stdout:-- Logs begin at Thu 2023-07-20 17:15:39 UTC. -- 2023-07-20T17:44:58.103 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:44:57 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-2[104225]: 2023-07-20T17:44:57.770+0000 7f63515d6640 -1 osd.2 0 log_to_monitors true 2023-07-20T17:44:58.103 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:44:58.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:58 smithi049 ceph-mon[96530]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:58.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:58.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:58.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:58.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:58 smithi049 ceph-mon[96530]: from='osd.2 [v2:172.21.15.97:6800/2945387274,v1:172.21.15.97:6801/2945387274]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T17:44:58.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:58 smithi049 ceph-mon[96530]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T17:44:58.381 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:58 smithi097 ceph-mon[98398]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:58.381 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:58.382 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:58.382 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:58.382 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:58 smithi097 ceph-mon[98398]: from='osd.2 [v2:172.21.15.97:6800/2945387274,v1:172.21.15.97:6801/2945387274]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T17:44:58.382 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:58 smithi097 ceph-mon[98398]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T17:44:58.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:58 smithi079 ceph-mon[98036]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:44:58.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:44:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:44:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:58 smithi079 ceph-mon[98036]: from='osd.2 [v2:172.21.15.97:6800/2945387274,v1:172.21.15.97:6801/2945387274]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T17:44:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:58 smithi079 ceph-mon[98036]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T17:44:59.767 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:44:59 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-2[104225]: 2023-07-20T17:44:59.611+0000 7f6346c4e700 -1 osd.2 0 waiting for initial osdmap 2023-07-20T17:44:59.767 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:44:59 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-2[104225]: 2023-07-20T17:44:59.614+0000 7f633f23d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:44:59.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:59 smithi097 ceph-mon[98398]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T17:44:59.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:59 smithi097 ceph-mon[98398]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T17:44:59.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:59 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:44:59.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:59 smithi097 ceph-mon[98398]: from='osd.2 [v2:172.21.15.97:6800/2945387274,v1:172.21.15.97:6801/2945387274]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:44:59.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:44:59 smithi097 ceph-mon[98398]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:44:59.780 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T17:44:59.781 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T17:44:59.781 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2023-07-20T17:44:59.781 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2023-07-20T17:44:59.781 INFO:teuthology.orchestra.run.smithi049.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0132087 s, 794 MB/s 2023-07-20T17:44:59.781 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 2023-07-20T17:44:59.781 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2023-07-20T17:44:59.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:59 smithi049 ceph-mon[96530]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T17:44:59.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:59 smithi049 ceph-mon[96530]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T17:44:59.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:59 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:44:59.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:59 smithi049 ceph-mon[96530]: from='osd.2 [v2:172.21.15.97:6800/2945387274,v1:172.21.15.97:6801/2945387274]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:44:59.925 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:44:59 smithi049 ceph-mon[96530]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:45:00.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:59 smithi079 ceph-mon[98036]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T17:45:00.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:59 smithi079 ceph-mon[98036]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T17:45:00.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:59 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:00.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:59 smithi079 ceph-mon[98036]: from='osd.2 [v2:172.21.15.97:6800/2945387274,v1:172.21.15.97:6801/2945387274]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:45:00.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:44:59 smithi079 ceph-mon[98036]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:45:00.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi049:vg_nvme/lv_3 2023-07-20T17:45:00.722 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:45:00.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:45:00.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:45:00.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T17:45:00.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:00.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:00.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:00.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:00.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:00.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:00.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:00.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:00.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:01.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:45:01.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:45:01.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T17:45:01.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:01.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:01.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:01.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:01.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:01.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:01.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:01.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:01.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:01.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:45:01.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:45:01.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T17:45:01.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:01.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:01.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:01.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:01.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:01.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:01.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:01.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:01.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:01.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:01 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:45:01.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:01 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:45:01.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:01 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi097 2023-07-20T17:45:01.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:01 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi097 to 17168M 2023-07-20T17:45:01.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:01 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:01.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:01 smithi049 ceph-mon[96530]: osd.2 [v2:172.21.15.97:6800/2945387274,v1:172.21.15.97:6801/2945387274] boot 2023-07-20T17:45:01.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:01 smithi049 ceph-mon[96530]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T17:45:01.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:01 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:01.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:01 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T17:45:02.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:01 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:45:02.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:01 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:45:02.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:01 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi097 2023-07-20T17:45:02.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:01 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi097 to 17168M 2023-07-20T17:45:02.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:01 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:02.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:01 smithi097 ceph-mon[98398]: osd.2 [v2:172.21.15.97:6800/2945387274,v1:172.21.15.97:6801/2945387274] boot 2023-07-20T17:45:02.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:01 smithi097 ceph-mon[98398]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T17:45:02.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:01 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:02.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:01 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T17:45:02.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.037+0000 7f590c13b700 1 -- 172.21.15.49:0/2865691757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5904100800 msgr2=0x7f5904100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.037+0000 7f590c13b700 1 --2- 172.21.15.49:0/2865691757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5904100800 0x7f5904100c20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f58f40099c0 tx=0x7f58f4009cd0 comp rx=0 tx=0).stop 2023-07-20T17:45:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.037+0000 7f590c13b700 1 -- 172.21.15.49:0/2865691757 shutdown_connections 2023-07-20T17:45:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.037+0000 7f590c13b700 1 --2- 172.21.15.49:0/2865691757 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59041023a0 0x7f5904108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.037+0000 7f590c13b700 1 --2- 172.21.15.49:0/2865691757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5904100800 0x7f5904100c20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:02.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.037+0000 7f590c13b700 1 --2- 172.21.15.49:0/2865691757 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5904101a00 0x7f5904101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.037+0000 7f590c13b700 1 -- 172.21.15.49:0/2865691757 >> 172.21.15.49:0/2865691757 conn(0x7f59040fbd60 msgr2=0x7f59040fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:45:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.037+0000 7f590c13b700 1 -- 172.21.15.49:0/2865691757 shutdown_connections 2023-07-20T17:45:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.037+0000 7f590c13b700 1 -- 172.21.15.49:0/2865691757 wait complete. 2023-07-20T17:45:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f590c13b700 1 Processor -- start 2023-07-20T17:45:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f590c13b700 1 -- start start 2023-07-20T17:45:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f590c13b700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904100800 0x7f59041964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f590c13b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5904101a00 0x7f5904196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f590c13b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59041023a0 0x7f590419da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f590c13b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590419e060 con 0x7f5904101a00 2023-07-20T17:45:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f590c13b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590419e1a0 con 0x7f59041023a0 2023-07-20T17:45:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f590c13b700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590419e330 con 0x7f5904100800 2023-07-20T17:45:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f59096d6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5904101a00 0x7f5904196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f59096d6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5904101a00 0x7f5904196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35646/0 (socket says 172.21.15.49:35646) 2023-07-20T17:45:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f59096d6700 1 -- 172.21.15.49:0/2346812869 learned_addr learned my addr 172.21.15.49:0/2346812869 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:45:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f5909ed7700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904100800 0x7f59041964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f59096d6700 1 -- 172.21.15.49:0/2346812869 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904100800 msgr2=0x7f59041964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f59096d6700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904100800 0x7f59041964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f59096d6700 1 -- 172.21.15.49:0/2346812869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59041023a0 msgr2=0x7f590419da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:02.067 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.038+0000 7f590a6d8700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59041023a0 0x7f590419da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:02.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.039+0000 7f59096d6700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59041023a0 0x7f590419da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:02.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.039+0000 7f59096d6700 1 -- 172.21.15.49:0/2346812869 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f40095f0 con 0x7f5904101a00 2023-07-20T17:45:02.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.039+0000 7f590a6d8700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59041023a0 0x7f590419da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T17:45:02.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.039+0000 7f59096d6700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5904101a00 0x7f5904196a10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f590000d980 tx=0x7f590000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:45:02.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.039+0000 7f58faffd700 1 -- 172.21.15.49:0/2346812869 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900009b40 con 0x7f5904101a00 2023-07-20T17:45:02.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.039+0000 7f590c13b700 1 -- 172.21.15.49:0/2346812869 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590419e610 con 0x7f5904101a00 2023-07-20T17:45:02.069 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.039+0000 7f58faffd700 1 -- 172.21.15.49:0/2346812869 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f590000b6e0 con 0x7f5904101a00 2023-07-20T17:45:02.069 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.039+0000 7f58faffd700 1 -- 172.21.15.49:0/2346812869 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590000f620 con 0x7f5904101a00 2023-07-20T17:45:02.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.039+0000 7f590c13b700 1 -- 172.21.15.49:0/2346812869 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590419eb10 con 0x7f5904101a00 2023-07-20T17:45:02.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.040+0000 7f590c13b700 1 -- 172.21.15.49:0/2346812869 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5904107e20 con 0x7f5904101a00 2023-07-20T17:45:02.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.041+0000 7f58faffd700 1 -- 172.21.15.49:0/2346812869 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f5900010460 con 0x7f5904101a00 2023-07-20T17:45:02.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.042+0000 7f58faffd700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58f00a10c0 0x7f58f00a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:02.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.042+0000 7f5909ed7700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58f00a10c0 0x7f58f00a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:02.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.042+0000 7f58faffd700 1 -- 172.21.15.49:0/2346812869 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7f59000cb3f0 con 0x7f5904101a00 2023-07-20T17:45:02.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.042+0000 7f5909ed7700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58f00a10c0 0x7f58f00a3580 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009990 tx=0x7f58f4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:45:02.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.045+0000 7f58faffd700 1 -- 172.21.15.49:0/2346812869 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5900095f60 con 0x7f5904101a00 2023-07-20T17:45:02.072 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:01 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:45:02.073 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:01 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:45:02.073 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:01 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi097 2023-07-20T17:45:02.073 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:01 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi097 to 17168M 2023-07-20T17:45:02.073 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:01 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:02.074 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:01 smithi079 ceph-mon[98036]: osd.2 [v2:172.21.15.97:6800/2945387274,v1:172.21.15.97:6801/2945387274] boot 2023-07-20T17:45:02.074 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:01 smithi079 ceph-mon[98036]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T17:45:02.074 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:01 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T17:45:02.074 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:01 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T17:45:02.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:02.248+0000 7f590c13b700 1 -- 172.21.15.49:0/2346812869 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5904060bb0 con 0x7f58f00a10c0 2023-07-20T17:45:02.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:02 smithi049 ceph-mon[96530]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:45:02.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T17:45:02.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:02 smithi049 ceph-mon[96530]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T17:45:02.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T17:45:02.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:45:02.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:45:02.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:03.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:02 smithi097 ceph-mon[98398]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:45:03.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T17:45:03.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:02 smithi097 ceph-mon[98398]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T17:45:03.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T17:45:03.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:45:03.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:45:03.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:03.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:02 smithi079 ceph-mon[98036]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T17:45:03.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T17:45:03.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:02 smithi079 ceph-mon[98036]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T17:45:03.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T17:45:03.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:45:03.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:45:03.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:03.473 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:45:03 smithi049 sudo[112621]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T17:45:03.473 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:45:03 smithi049 sudo[112621]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:03.473 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:45:03 smithi049 sudo[112621]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:03.473 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:45:03 smithi049 sudo[112621]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:03.642 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:03.640+0000 7f58faffd700 1 -- 172.21.15.49:0/2346812869 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5900096330 con 0x7f5904101a00 2023-07-20T17:45:03.739 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:45:03 smithi049 sudo[112659]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T17:45:03.739 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:45:03 smithi049 sudo[112659]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:03.739 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:45:03 smithi049 sudo[112659]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:03.739 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:45:03 smithi049 sudo[112659]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:03.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:03 smithi049 ceph-mon[96530]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:45:03.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T17:45:03.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:03 smithi049 ceph-mon[96530]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T17:45:03.917 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:03 smithi079 ceph-mon[98036]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:45:03.917 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T17:45:03.917 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:03 smithi079 ceph-mon[98036]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T17:45:04.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:03 smithi097 ceph-mon[98398]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:45:04.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T17:45:04.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:03 smithi097 ceph-mon[98398]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T17:45:04.169 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:45:03 smithi079 sudo[105423]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T17:45:04.169 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:45:03 smithi079 sudo[105423]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:04.169 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:45:03 smithi079 sudo[105423]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:04.169 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:45:03 smithi079 sudo[105423]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:04.566 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:45:04 smithi079 sudo[105426]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T17:45:04.566 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:45:04 smithi079 sudo[105426]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:04.567 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:45:04 smithi079 sudo[105426]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:04.567 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:45:04 smithi079 sudo[105426]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:04.691 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:45:04 smithi097 sudo[106456]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T17:45:04.692 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:45:04 smithi097 sudo[106456]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:04.692 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:45:04 smithi097 sudo[106456]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:04.692 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:45:04 smithi097 sudo[106456]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:04.692 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:04 smithi097 ceph-mon[98398]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:04.693 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:04 smithi097 ceph-mon[98398]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T17:45:04.693 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:04 smithi097 ceph-mon[98398]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T17:45:04.693 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:04 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:04.693 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:04 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:04.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:04 smithi049 ceph-mon[96530]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:04.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:04 smithi049 ceph-mon[96530]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T17:45:04.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:04 smithi049 ceph-mon[96530]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T17:45:04.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:04 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:04.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:04 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:05.016 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:45:04 smithi097 sudo[106459]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T17:45:05.017 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:45:04 smithi097 sudo[106459]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:05.017 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:45:04 smithi097 sudo[106459]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:05.017 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:45:04 smithi097 sudo[106459]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:05.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:04 smithi079 ceph-mon[98036]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:05.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:04 smithi079 ceph-mon[98036]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T17:45:05.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:04 smithi079 ceph-mon[98036]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T17:45:05.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:04 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:05.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:04 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:05.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:05 smithi049 sudo[112787]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T17:45:05.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:05 smithi049 sudo[112787]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:05.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:05 smithi049 sudo[112787]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:05.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:05 smithi049 sudo[112787]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:05.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:05 smithi049 sudo[112840]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T17:45:05.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:05 smithi049 sudo[112840]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:05.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:05 smithi049 sudo[112840]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:05.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:05 smithi049 sudo[112840]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:06.105 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:05 smithi097 sudo[106462]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T17:45:06.106 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:05 smithi097 sudo[106462]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:06.106 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:05 smithi097 sudo[106462]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:06.106 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:05 smithi097 sudo[106462]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:06.364 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/338296160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ba1972a-2bf4-4602-900c-11b15dcf6027"}]: dispatch 2023-07-20T17:45:06.364 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/338296160' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ba1972a-2bf4-4602-900c-11b15dcf6027"}]': finished 2023-07-20T17:45:06.364 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T17:45:06.364 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:06.365 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:06.365 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T17:45:06.365 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1058782520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:45:06.365 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T17:45:06.365 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:45:06.365 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:45:06.365 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:45:06.366 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:45:06.366 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:45:06.366 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:45:06.366 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 sudo[105429]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T17:45:06.366 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 sudo[105429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:06.366 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 sudo[105429]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:06.366 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 sudo[105429]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:06.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/338296160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ba1972a-2bf4-4602-900c-11b15dcf6027"}]: dispatch 2023-07-20T17:45:06.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/338296160' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ba1972a-2bf4-4602-900c-11b15dcf6027"}]': finished 2023-07-20T17:45:06.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T17:45:06.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:06.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:06.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T17:45:06.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1058782520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:45:06.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T17:45:06.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:45:06.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:45:06.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:45:06.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:45:06.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:45:06.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:45:06.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 sudo[106465]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T17:45:06.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 sudo[106465]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:06.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 sudo[106465]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:06.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 sudo[106465]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:06.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/338296160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ba1972a-2bf4-4602-900c-11b15dcf6027"}]: dispatch 2023-07-20T17:45:06.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/338296160' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ba1972a-2bf4-4602-900c-11b15dcf6027"}]': finished 2023-07-20T17:45:06.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T17:45:06.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:06.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:06.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T17:45:06.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1058782520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:45:06.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T17:45:06.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:45:06.521 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:45:06.521 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:45:06.521 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:45:06.522 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:45:06.522 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:45:06.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 sudo[105432]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T17:45:06.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 sudo[105432]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T17:45:06.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 sudo[105432]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T17:45:06.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:06 smithi079 sudo[105432]: pam_unix(sudo:session): session closed for user root 2023-07-20T17:45:07.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:07 smithi049 ceph-mon[96530]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:07.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:07 smithi049 ceph-mon[96530]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T17:45:07.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:07 smithi049 ceph-mon[96530]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T17:45:07.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:07 smithi049 ceph-mon[96530]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T17:45:07.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:07 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:45:07.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:07 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:45:07.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:07 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:45:07.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:07 smithi049 ceph-mon[96530]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T17:45:07.516 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:07 smithi097 ceph-mon[98398]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:07.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:07 smithi097 ceph-mon[98398]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T17:45:07.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:07 smithi097 ceph-mon[98398]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T17:45:07.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:07 smithi097 ceph-mon[98398]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T17:45:07.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:07 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:45:07.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:07 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:45:07.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:07 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:45:07.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:07 smithi097 ceph-mon[98398]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T17:45:07.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:07 smithi079 ceph-mon[98036]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:07.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:07 smithi079 ceph-mon[98036]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T17:45:07.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:07 smithi079 ceph-mon[98036]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T17:45:07.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:07 smithi079 ceph-mon[98036]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T17:45:07.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:07 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T17:45:07.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:07 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T17:45:07.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:07 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T17:45:07.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:07 smithi079 ceph-mon[98036]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T17:45:08.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:08 smithi049 ceph-mon[96530]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:09.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:08 smithi097 ceph-mon[98398]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:09.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:08 smithi079 ceph-mon[98036]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:10.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:10 smithi049 ceph-mon[96530]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:11.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:10 smithi097 ceph-mon[98398]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:11.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:10 smithi079 ceph-mon[98036]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:11.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:11 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T17:45:11.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:11 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:12.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:11 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T17:45:12.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:11 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:12.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:11 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T17:45:12.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:11 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:12.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:12 smithi049 ceph-mon[96530]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-07-20T17:45:12.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:12 smithi049 ceph-mon[96530]: Deploying daemon osd.3 on smithi049 2023-07-20T17:45:13.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:12 smithi097 ceph-mon[98398]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-07-20T17:45:13.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:12 smithi097 ceph-mon[98398]: Deploying daemon osd.3 on smithi049 2023-07-20T17:45:13.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:12 smithi079 ceph-mon[98036]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-07-20T17:45:13.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:12 smithi079 ceph-mon[98036]: Deploying daemon osd.3 on smithi049 2023-07-20T17:45:14.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:14 smithi049 ceph-mon[96530]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-07-20T17:45:15.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:14 smithi097 ceph-mon[98398]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-07-20T17:45:15.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:14 smithi079 ceph-mon[98036]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-07-20T17:45:16.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:16 smithi049 ceph-mon[96530]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-07-20T17:45:16.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:16 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:16.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:16 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:16.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:16 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:17.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:16 smithi097 ceph-mon[98398]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-07-20T17:45:17.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:16 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:17.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:16 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:17.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:16 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:17.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:16 smithi079 ceph-mon[98036]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-07-20T17:45:17.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:16 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:17.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:16 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:17.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:16 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:18.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.500+0000 7f58faffd700 1 -- 172.21.15.49:0/2346812869 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5904060bb0 con 0x7f58f00a10c0 2023-07-20T17:45:18.501 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 3 on host 'smithi049' 2023-07-20T17:45:18.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 -- 172.21.15.49:0/2346812869 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58f00a10c0 msgr2=0x7f58f00a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:18.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58f00a10c0 0x7f58f00a3580 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009990 tx=0x7f58f4005fd0 comp rx=0 tx=0).stop 2023-07-20T17:45:18.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 -- 172.21.15.49:0/2346812869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5904101a00 msgr2=0x7f5904196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:18.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5904101a00 0x7f5904196a10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f590000d980 tx=0x7f590000dd40 comp rx=0 tx=0).stop 2023-07-20T17:45:18.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 -- 172.21.15.49:0/2346812869 shutdown_connections 2023-07-20T17:45:18.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58f00a10c0 0x7f58f00a3580 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5904100800 0x7f59041964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59041023a0 0x7f590419da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 --2- 172.21.15.49:0/2346812869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5904101a00 0x7f5904196a10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 -- 172.21.15.49:0/2346812869 >> 172.21.15.49:0/2346812869 conn(0x7f59040fbd60 msgr2=0x7f5904104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 -- 172.21.15.49:0/2346812869 shutdown_connections 2023-07-20T17:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:45:18.502+0000 7f590c13b700 1 -- 172.21.15.49:0/2346812869 wait complete. 2023-07-20T17:45:19.016 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:18 smithi097 ceph-mon[98398]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-20T17:45:19.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:19.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:19.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:19.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:18 smithi049 ceph-mon[96530]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-20T17:45:19.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:19.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:19.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:19.033 DEBUG:teuthology.orchestra.run.smithi049:osd.3> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.3.service 2023-07-20T17:45:19.035 INFO:tasks.cephadm:Deploying osd.4 on smithi079 with /dev/vg_nvme/lv_3... 2023-07-20T17:45:19.036 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T17:45:19.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:18 smithi079 ceph-mon[98036]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-20T17:45:19.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:19.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:19.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:19.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:19.071 INFO:journalctl@ceph.osd.3.smithi049.stdout:-- Logs begin at Thu 2023-07-20 17:16:49 UTC. -- 2023-07-20T17:45:19.595 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T17:45:20.239 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jul 20 17:45:19 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-3[114991]: 2023-07-20T17:45:19.768+0000 7fdcb9362640 -1 osd.3 0 log_to_monitors true 2023-07-20T17:45:21.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:20 smithi097 ceph-mon[98398]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-20T17:45:21.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:20 smithi097 ceph-mon[98398]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T17:45:21.050 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T17:45:21.050 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T17:45:21.050 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2023-07-20T17:45:21.050 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2023-07-20T17:45:21.051 INFO:teuthology.orchestra.run.smithi079.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130687 s, 802 MB/s 2023-07-20T17:45:21.051 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2023-07-20T17:45:21.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:20 smithi079 ceph-mon[98036]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-20T17:45:21.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:20 smithi079 ceph-mon[98036]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T17:45:21.205 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:20 smithi049 ceph-mon[96530]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-20T17:45:21.205 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:20 smithi049 ceph-mon[96530]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T17:45:21.445 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi079:vg_nvme/lv_3 2023-07-20T17:45:22.034 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T17:45:22.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T17:45:22.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T17:45:22.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:22.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:45:22.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:22.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:22.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:22.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:22.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:22.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:22.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:22.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:22.239 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-3[114991]: 2023-07-20T17:45:21.772+0000 7fdcae9da700 -1 osd.3 0 waiting for initial osdmap 2023-07-20T17:45:22.239 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-3[114991]: 2023-07-20T17:45:21.777+0000 7fdcaa7d0700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:45:22.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T17:45:22.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T17:45:22.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:22.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:45:22.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:22.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:22.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:22.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:22.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:22.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:22.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:22.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:22.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T17:45:22.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T17:45:22.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:22.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:45:22.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:22.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:22.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:22.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:22.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:22.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:22.271 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:22.271 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:23.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:22 smithi079 ceph-mon[98036]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:23.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:22 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi049 2023-07-20T17:45:23.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:22 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi049 to 8584M 2023-07-20T17:45:23.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:22 smithi079 ceph-mon[98036]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:45:23.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:22 smithi079 ceph-mon[98036]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T17:45:23.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:22 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:23.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:22 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:23.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:22 smithi079 ceph-mon[98036]: osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359] boot 2023-07-20T17:45:23.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:22 smithi079 ceph-mon[98036]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T17:45:23.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:22 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:23.116 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.115+0000 7f0990dae700 1 -- 172.21.15.79:0/1634677641 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f098c0ff700 msgr2=0x7f098c0ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:23.116 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.115+0000 7f0990dae700 1 --2- 172.21.15.79:0/1634677641 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f098c0ff700 0x7f098c0ffb20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f09800099c0 tx=0x7f0980009cd0 comp rx=0 tx=0).stop 2023-07-20T17:45:23.116 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.115+0000 7f0990dae700 1 -- 172.21.15.79:0/1634677641 shutdown_connections 2023-07-20T17:45:23.116 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.115+0000 7f0990dae700 1 --2- 172.21.15.79:0/1634677641 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f098c1012a0 0x7f098c107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:23.117 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.115+0000 7f0990dae700 1 --2- 172.21.15.79:0/1634677641 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f098c0ff700 0x7f098c0ffb20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:23.117 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.115+0000 7f0990dae700 1 --2- 172.21.15.79:0/1634677641 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f098c100900 0x7f098c100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:23.117 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.115+0000 7f0990dae700 1 -- 172.21.15.79:0/1634677641 >> 172.21.15.79:0/1634677641 conn(0x7f098c0fac80 msgr2=0x7f098c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:45:23.117 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.115+0000 7f0990dae700 1 -- 172.21.15.79:0/1634677641 shutdown_connections 2023-07-20T17:45:23.117 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.115+0000 7f0990dae700 1 -- 172.21.15.79:0/1634677641 wait complete. 2023-07-20T17:45:23.119 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f0990dae700 1 Processor -- start 2023-07-20T17:45:23.119 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f0990dae700 1 -- start start 2023-07-20T17:45:23.119 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f0990dae700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f098c0ff700 0x7f098c1964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:23.119 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f0990dae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f098c100900 0x7f098c1969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:23.119 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f0990dae700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f098c1012a0 0x7f098c19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:23.119 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f0990dae700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c19e030 con 0x7f098c100900 2023-07-20T17:45:23.119 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f0990dae700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c19e170 con 0x7f098c1012a0 2023-07-20T17:45:23.120 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f0990dae700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c19e300 con 0x7f098c0ff700 2023-07-20T17:45:23.120 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f098ad9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f098c1012a0 0x7f098c19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:23.120 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f098ad9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f098c1012a0 0x7f098c19da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47824/0 (socket says 172.21.15.79:47824) 2023-07-20T17:45:23.120 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f098ad9d700 1 -- 172.21.15.79:0/2746077597 learned_addr learned my addr 172.21.15.79:0/2746077597 (peer_addr_for_me v2:172.21.15.79:0/0) 2023-07-20T17:45:23.120 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f098a59c700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f098c0ff700 0x7f098c1964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:23.120 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f0989d9b700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f098c100900 0x7f098c1969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:23.120 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f098ad9d700 1 -- 172.21.15.79:0/2746077597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f098c0ff700 msgr2=0x7f098c1964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:23.121 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f098ad9d700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f098c0ff700 0x7f098c1964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:23.121 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f098ad9d700 1 -- 172.21.15.79:0/2746077597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f098c100900 msgr2=0x7f098c1969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:23.121 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f098ad9d700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f098c100900 0x7f098c1969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:23.121 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f098ad9d700 1 -- 172.21.15.79:0/2746077597 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09800095f0 con 0x7f098c1012a0 2023-07-20T17:45:23.121 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.116+0000 7f0989d9b700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f098c100900 0x7f098c1969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T17:45:23.122 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.117+0000 7f098ad9d700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f098c1012a0 0x7f098c19da60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f098400ada0 tx=0x7f098400c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:45:23.122 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.117+0000 7f097b7fe700 1 -- 172.21.15.79:0/2746077597 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098400f070 con 0x7f098c1012a0 2023-07-20T17:45:23.122 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.117+0000 7f0990dae700 1 -- 172.21.15.79:0/2746077597 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098c19e5e0 con 0x7f098c1012a0 2023-07-20T17:45:23.122 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.117+0000 7f097b7fe700 1 -- 172.21.15.79:0/2746077597 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0984007b30 con 0x7f098c1012a0 2023-07-20T17:45:23.123 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.117+0000 7f097b7fe700 1 -- 172.21.15.79:0/2746077597 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09840053e0 con 0x7f098c1012a0 2023-07-20T17:45:23.123 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.117+0000 7f0990dae700 1 -- 172.21.15.79:0/2746077597 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098c19ea90 con 0x7f098c1012a0 2023-07-20T17:45:23.123 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.118+0000 7f0990dae700 1 -- 172.21.15.79:0/2746077597 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098c10af00 con 0x7f098c1012a0 2023-07-20T17:45:23.125 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.119+0000 7f097b7fe700 1 -- 172.21.15.79:0/2746077597 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f098400c980 con 0x7f098c1012a0 2023-07-20T17:45:23.125 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.119+0000 7f097b7fe700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f09740a1150 0x7f09740a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:23.125 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.120+0000 7f097b7fe700 1 -- 172.21.15.79:0/2746077597 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 3702+0+0 (secure 0 0 0) 0x7f09840cace0 con 0x7f098c1012a0 2023-07-20T17:45:23.126 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.120+0000 7f098a59c700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f09740a1150 0x7f09740a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:23.126 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.120+0000 7f098a59c700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f09740a1150 0x7f09740a3610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0980009f50 tx=0x7f098000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:45:23.126 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.124+0000 7f097b7fe700 1 -- 172.21.15.79:0/2746077597 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f09840956c0 con 0x7f098c1012a0 2023-07-20T17:45:23.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:22 smithi049 ceph-mon[96530]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:23.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:22 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi049 2023-07-20T17:45:23.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:22 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi049 to 8584M 2023-07-20T17:45:23.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:22 smithi049 ceph-mon[96530]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:45:23.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:22 smithi049 ceph-mon[96530]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T17:45:23.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:22 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:23.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:22 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:23.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:22 smithi049 ceph-mon[96530]: osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359] boot 2023-07-20T17:45:23.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:22 smithi049 ceph-mon[96530]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T17:45:23.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:22 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:23.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:22 smithi097 ceph-mon[98398]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T17:45:23.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:22 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi049 2023-07-20T17:45:23.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:22 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi049 to 8584M 2023-07-20T17:45:23.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:22 smithi097 ceph-mon[98398]: from='osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:45:23.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:22 smithi097 ceph-mon[98398]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T17:45:23.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:22 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:23.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:22 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:23.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:22 smithi097 ceph-mon[98398]: osd.3 [v2:172.21.15.49:6810/178015359,v1:172.21.15.49:6811/178015359] boot 2023-07-20T17:45:23.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:22 smithi097 ceph-mon[98398]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T17:45:23.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:22 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T17:45:23.315 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:23.314+0000 7f0990dae700 1 -- 172.21.15.79:0/2746077597 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f098c068980 con 0x7f09740a1150 2023-07-20T17:45:24.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:23 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:45:24.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:23 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:45:24.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:45:24.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:45:24.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:24.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:23 smithi079 ceph-mon[98036]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T17:45:24.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:23 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:45:24.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:23 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:45:24.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:45:24.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:45:24.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:24.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:23 smithi049 ceph-mon[96530]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T17:45:24.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:23 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:45:24.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:23 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:45:24.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:45:24.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:45:24.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:24.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:23 smithi097 ceph-mon[98398]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T17:45:24.896 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:24 smithi079 ceph-mon[98036]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:24.897 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:24 smithi079 ceph-mon[98036]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:45:24.897 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:24 smithi079 ceph-mon[98036]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T17:45:25.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:24 smithi049 ceph-mon[96530]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:25.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:24 smithi049 ceph-mon[96530]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:45:25.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:24 smithi049 ceph-mon[96530]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T17:45:25.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:24 smithi097 ceph-mon[98398]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:25.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:24 smithi097 ceph-mon[98398]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:45:25.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:24 smithi097 ceph-mon[98398]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T17:45:26.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:26 smithi049 ceph-mon[96530]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:26.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:26 smithi049 ceph-mon[96530]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T17:45:26.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:26 smithi049 ceph-mon[96530]: from='client.? 172.21.15.79:0/1565973962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a167452b-f627-4aae-aecd-e513a5649235"}]: dispatch 2023-07-20T17:45:26.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:26 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a167452b-f627-4aae-aecd-e513a5649235"}]: dispatch 2023-07-20T17:45:26.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:26 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a167452b-f627-4aae-aecd-e513a5649235"}]': finished 2023-07-20T17:45:26.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:26 smithi049 ceph-mon[96530]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T17:45:26.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:26.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:26 smithi049 ceph-mon[96530]: from='client.? 172.21.15.79:0/2074518871' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:45:26.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:26 smithi097 ceph-mon[98398]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:26.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:26 smithi097 ceph-mon[98398]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T17:45:26.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:26 smithi097 ceph-mon[98398]: from='client.? 172.21.15.79:0/1565973962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a167452b-f627-4aae-aecd-e513a5649235"}]: dispatch 2023-07-20T17:45:26.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:26 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a167452b-f627-4aae-aecd-e513a5649235"}]: dispatch 2023-07-20T17:45:26.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:26 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a167452b-f627-4aae-aecd-e513a5649235"}]': finished 2023-07-20T17:45:26.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:26 smithi097 ceph-mon[98398]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T17:45:26.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:26.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:26 smithi097 ceph-mon[98398]: from='client.? 172.21.15.79:0/2074518871' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:45:26.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:26 smithi079 ceph-mon[98036]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:26.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:26 smithi079 ceph-mon[98036]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T17:45:26.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:26 smithi079 ceph-mon[98036]: from='client.? 172.21.15.79:0/1565973962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a167452b-f627-4aae-aecd-e513a5649235"}]: dispatch 2023-07-20T17:45:26.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:26 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a167452b-f627-4aae-aecd-e513a5649235"}]: dispatch 2023-07-20T17:45:26.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:26 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a167452b-f627-4aae-aecd-e513a5649235"}]': finished 2023-07-20T17:45:26.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:26 smithi079 ceph-mon[98036]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T17:45:26.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:26.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:26 smithi079 ceph-mon[98036]: from='client.? 172.21.15.79:0/2074518871' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:45:28.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:28 smithi049 ceph-mon[96530]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T17:45:28.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:28 smithi097 ceph-mon[98398]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T17:45:28.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:28 smithi079 ceph-mon[98036]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T17:45:30.728 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:30 smithi079 ceph-mon[98036]: pgmap v97: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-07-20T17:45:30.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:30 smithi049 ceph-mon[96530]: pgmap v97: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-07-20T17:45:30.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:30 smithi097 ceph-mon[98398]: pgmap v97: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-07-20T17:45:32.686 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:32 smithi079 ceph-mon[98036]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-07-20T17:45:32.686 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:32 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T17:45:32.686 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:32 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:32.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:32 smithi049 ceph-mon[96530]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-07-20T17:45:32.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:32 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T17:45:32.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:32 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:32.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:32 smithi097 ceph-mon[98398]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-07-20T17:45:32.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:32 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T17:45:32.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:32 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:33.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:33 smithi049 ceph-mon[96530]: Deploying daemon osd.4 on smithi079 2023-07-20T17:45:33.766 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:33 smithi097 ceph-mon[98398]: Deploying daemon osd.4 on smithi079 2023-07-20T17:45:33.777 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:33 smithi079 ceph-mon[98036]: Deploying daemon osd.4 on smithi079 2023-07-20T17:45:34.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:34 smithi049 ceph-mon[96530]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T17:45:34.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:34 smithi097 ceph-mon[98398]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T17:45:34.809 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:34 smithi079 ceph-mon[98036]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T17:45:36.646 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:36 smithi079 ceph-mon[98036]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:36.646 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:36.646 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:36.646 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:36.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:36 smithi049 ceph-mon[96530]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:36.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:36.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:36.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:36.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:36 smithi097 ceph-mon[98398]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:36.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:36.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:36.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:38.737 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:38 smithi079 ceph-mon[98036]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:38.737 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:38.738 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:38.738 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:38.738 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:38.738 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:38.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:38 smithi049 ceph-mon[96530]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:38.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:38.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:38.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:38.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:38.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:38.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:38 smithi097 ceph-mon[98398]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:38.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:38.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:38.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:38.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:38.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:38.977 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.977+0000 7f097b7fe700 1 -- 172.21.15.79:0/2746077597 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f098c068980 con 0x7f09740a1150 2023-07-20T17:45:38.978 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 4 on host 'smithi079' 2023-07-20T17:45:38.980 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.979+0000 7f0990dae700 1 -- 172.21.15.79:0/2746077597 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f09740a1150 msgr2=0x7f09740a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:38.980 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.979+0000 7f0990dae700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f09740a1150 0x7f09740a3610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0980009f50 tx=0x7f098000b540 comp rx=0 tx=0).stop 2023-07-20T17:45:38.980 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.979+0000 7f0990dae700 1 -- 172.21.15.79:0/2746077597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f098c1012a0 msgr2=0x7f098c19da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:38.981 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.979+0000 7f0990dae700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f098c1012a0 0x7f098c19da60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f098400ada0 tx=0x7f098400c330 comp rx=0 tx=0).stop 2023-07-20T17:45:38.981 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.979+0000 7f0990dae700 1 -- 172.21.15.79:0/2746077597 shutdown_connections 2023-07-20T17:45:38.981 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.979+0000 7f0990dae700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f09740a1150 0x7f09740a3610 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:38.981 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.979+0000 7f0990dae700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f098c0ff700 0x7f098c1964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:38.981 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.979+0000 7f0990dae700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f098c1012a0 0x7f098c19da60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:38.981 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.979+0000 7f0990dae700 1 --2- 172.21.15.79:0/2746077597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f098c100900 0x7f098c1969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:38.981 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.979+0000 7f0990dae700 1 -- 172.21.15.79:0/2746077597 >> 172.21.15.79:0/2746077597 conn(0x7f098c0fac80 msgr2=0x7f098c068270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:45:38.982 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.980+0000 7f0990dae700 1 -- 172.21.15.79:0/2746077597 shutdown_connections 2023-07-20T17:45:38.982 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:45:38.980+0000 7f0990dae700 1 -- 172.21.15.79:0/2746077597 wait complete. 2023-07-20T17:45:39.324 DEBUG:teuthology.orchestra.run.smithi079:osd.4> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.4.service 2023-07-20T17:45:39.326 INFO:tasks.cephadm:Deploying osd.5 on smithi097 with /dev/vg_nvme/lv_3... 2023-07-20T17:45:39.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T17:45:39.355 INFO:journalctl@ceph.osd.4.smithi079.stdout:-- Logs begin at Thu 2023-07-20 17:15:42 UTC. -- 2023-07-20T17:45:39.887 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:45:40.189 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:39 smithi079 ceph-mon[98036]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:40.189 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:39 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:40.189 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:39 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:40.189 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:39 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:40.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:39 smithi049 ceph-mon[96530]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:40.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:39 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:40.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:39 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:40.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:39 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:40.248 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:39 smithi097 ceph-mon[98398]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:40.249 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:39 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:40.249 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:39 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:40.249 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:39 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:40.535 INFO:journalctl@ceph.osd.4.smithi079.stdout:Jul 20 17:45:40 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-4[108622]: 2023-07-20T17:45:40.187+0000 7fde5381b640 -1 osd.4 0 log_to_monitors true 2023-07-20T17:45:41.237 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:40 smithi079 ceph-mon[98036]: from='osd.4 [v2:172.21.15.79:6808/4037011752,v1:172.21.15.79:6809/4037011752]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T17:45:41.237 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:40 smithi079 ceph-mon[98036]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T17:45:41.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:40 smithi049 ceph-mon[96530]: from='osd.4 [v2:172.21.15.79:6808/4037011752,v1:172.21.15.79:6809/4037011752]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T17:45:41.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:40 smithi049 ceph-mon[96530]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T17:45:41.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:40 smithi097 ceph-mon[98398]: from='osd.4 [v2:172.21.15.79:6808/4037011752,v1:172.21.15.79:6809/4037011752]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T17:45:41.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:40 smithi097 ceph-mon[98398]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T17:45:41.415 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T17:45:41.415 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T17:45:41.415 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2023-07-20T17:45:41.415 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2023-07-20T17:45:41.416 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01285 s, 816 MB/s 2023-07-20T17:45:41.416 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2023-07-20T17:45:41.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi097:vg_nvme/lv_3 2023-07-20T17:45:42.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:42.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T17:45:42.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T17:45:42.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:42.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='osd.4 [v2:172.21.15.79:6808/4037011752,v1:172.21.15.79:6809/4037011752]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:45:42.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:45:42.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi079 2023-07-20T17:45:42.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi079 to 8584M 2023-07-20T17:45:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:41 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:42.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T17:45:42.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T17:45:42.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:42.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='osd.4 [v2:172.21.15.79:6808/4037011752,v1:172.21.15.79:6809/4037011752]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:45:42.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:45:42.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi079 2023-07-20T17:45:42.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:42.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:42.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi079 to 8584M 2023-07-20T17:45:42.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:42.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:42.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:41 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.316 INFO:journalctl@ceph.osd.4.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-4[108622]: 2023-07-20T17:45:41.974+0000 7fde48e93700 -1 osd.4 0 waiting for initial osdmap 2023-07-20T17:45:42.317 INFO:journalctl@ceph.osd.4.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-4[108622]: 2023-07-20T17:45:41.979+0000 7fde42c85700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:45:42.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:42.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T17:45:42.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T17:45:42.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:42.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='osd.4 [v2:172.21.15.79:6808/4037011752,v1:172.21.15.79:6809/4037011752]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:45:42.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:45:42.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi079 2023-07-20T17:45:42.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:45:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi079 to 8584M 2023-07-20T17:45:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:45:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:41 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:42.392 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:45:43.238 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:42 smithi097 ceph-mon[98398]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:45:43.238 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:42 smithi097 ceph-mon[98398]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T17:45:43.238 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:43.238 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:43.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:42 smithi049 ceph-mon[96530]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:45:43.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:42 smithi049 ceph-mon[96530]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T17:45:43.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:43.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:43.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:42 smithi079 ceph-mon[98036]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:45:43.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:42 smithi079 ceph-mon[98036]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T17:45:43.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:43.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:43.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.719+0000 7fcd1f32d700 1 -- 172.21.15.97:0/2836999891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd180ff700 msgr2=0x7fcd180ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:43.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.719+0000 7fcd1f32d700 1 --2- 172.21.15.97:0/2836999891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd180ff700 0x7fcd180ffb20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcd08009990 tx=0x7fcd08009ca0 comp rx=0 tx=0).stop 2023-07-20T17:45:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.719+0000 7fcd1f32d700 1 -- 172.21.15.97:0/2836999891 shutdown_connections 2023-07-20T17:45:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.719+0000 7fcd1f32d700 1 --2- 172.21.15.97:0/2836999891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd180ff700 0x7fcd180ffb20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.719+0000 7fcd1f32d700 1 --2- 172.21.15.97:0/2836999891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd181012a0 0x7fcd18107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.719+0000 7fcd1f32d700 1 --2- 172.21.15.97:0/2836999891 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd18100900 0x7fcd18100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.719+0000 7fcd1f32d700 1 -- 172.21.15.97:0/2836999891 >> 172.21.15.97:0/2836999891 conn(0x7fcd180fac80 msgr2=0x7fcd180fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:45:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.719+0000 7fcd1f32d700 1 -- 172.21.15.97:0/2836999891 shutdown_connections 2023-07-20T17:45:43.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.719+0000 7fcd1f32d700 1 -- 172.21.15.97:0/2836999891 wait complete. 2023-07-20T17:45:43.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1f32d700 1 Processor -- start 2023-07-20T17:45:43.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1f32d700 1 -- start start 2023-07-20T17:45:43.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1f32d700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd180ff700 0x7fcd181953c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:43.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1f32d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd18100900 0x7fcd18195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:43.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1f32d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd181012a0 0x7fcd1819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:43.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1f32d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1819cf50 con 0x7fcd181012a0 2023-07-20T17:45:43.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1f32d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd18068230 con 0x7fcd18100900 2023-07-20T17:45:43.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1f32d700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd180683f0 con 0x7fcd180ff700 2023-07-20T17:45:43.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1d0c9700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd180ff700 0x7fcd181953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:43.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1d0c9700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd180ff700 0x7fcd181953c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60742/0 (socket says 172.21.15.97:60742) 2023-07-20T17:45:43.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1d0c9700 1 -- 172.21.15.97:0/1420531591 learned_addr learned my addr 172.21.15.97:0/1420531591 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-07-20T17:45:43.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1c8c8700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd18100900 0x7fcd18195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:43.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1d8ca700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd181012a0 0x7fcd1819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:43.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1d0c9700 1 -- 172.21.15.97:0/1420531591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd18100900 msgr2=0x7fcd18195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:43.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1d0c9700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd18100900 0x7fcd18195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:43.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1d0c9700 1 -- 172.21.15.97:0/1420531591 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd181012a0 msgr2=0x7fcd1819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:45:43.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1d0c9700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd181012a0 0x7fcd1819c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:45:43.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.720+0000 7fcd1d0c9700 1 -- 172.21.15.97:0/1420531591 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd080095f0 con 0x7fcd180ff700 2023-07-20T17:45:43.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.721+0000 7fcd1d0c9700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd180ff700 0x7fcd181953c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcd08005b40 tx=0x7fcd0800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:45:43.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.721+0000 7fcd0e7fc700 1 -- 172.21.15.97:0/1420531591 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0801d070 con 0x7fcd180ff700 2023-07-20T17:45:43.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.721+0000 7fcd0e7fc700 1 -- 172.21.15.97:0/1420531591 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd08003e20 con 0x7fcd180ff700 2023-07-20T17:45:43.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.721+0000 7fcd1f32d700 1 -- 172.21.15.97:0/1420531591 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd18068670 con 0x7fcd180ff700 2023-07-20T17:45:43.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.721+0000 7fcd0e7fc700 1 -- 172.21.15.97:0/1420531591 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd08005680 con 0x7fcd180ff700 2023-07-20T17:45:43.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.721+0000 7fcd1f32d700 1 -- 172.21.15.97:0/1420531591 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd18068b60 con 0x7fcd180ff700 2023-07-20T17:45:43.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.722+0000 7fcd1f32d700 1 -- 172.21.15.97:0/1420531591 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1810af20 con 0x7fcd180ff700 2023-07-20T17:45:43.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.723+0000 7fcd0e7fc700 1 -- 172.21.15.97:0/1420531591 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcd0800fc80 con 0x7fcd180ff700 2023-07-20T17:45:43.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.723+0000 7fcd0e7fc700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fcd040a11d0 0x7fcd040a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:45:43.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.723+0000 7fcd0e7fc700 1 -- 172.21.15.97:0/1420531591 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fcd080ccb20 con 0x7fcd180ff700 2023-07-20T17:45:43.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.723+0000 7fcd1c8c8700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fcd040a11d0 0x7fcd040a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:45:43.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.724+0000 7fcd1c8c8700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fcd040a11d0 0x7fcd040a3690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcd14005950 tx=0x7fcd1400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:45:43.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.727+0000 7fcd0e7fc700 1 -- 172.21.15.97:0/1420531591 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcd08097370 con 0x7fcd180ff700 2023-07-20T17:45:43.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:45:43.918+0000 7fcd1f32d700 1 -- 172.21.15.97:0/1420531591 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd18060bb0 con 0x7fcd040a11d0 2023-07-20T17:45:44.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:43 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:45:44.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:43 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:45:44.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:43 smithi049 ceph-mon[96530]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:44.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:43 smithi049 ceph-mon[96530]: osd.4 [v2:172.21.15.79:6808/4037011752,v1:172.21.15.79:6809/4037011752] boot 2023-07-20T17:45:44.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:43 smithi049 ceph-mon[96530]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T17:45:44.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:43 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:44.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:43 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:45:44.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:43 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:45:44.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:43 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:44.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:43 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:45:44.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:43 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:45:44.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:43 smithi097 ceph-mon[98398]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:44.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:43 smithi097 ceph-mon[98398]: osd.4 [v2:172.21.15.79:6808/4037011752,v1:172.21.15.79:6809/4037011752] boot 2023-07-20T17:45:44.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:43 smithi097 ceph-mon[98398]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T17:45:44.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:43 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:44.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:43 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:45:44.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:43 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:45:44.271 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:43 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:44.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:43 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:45:44.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:43 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:45:44.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:43 smithi079 ceph-mon[98036]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T17:45:44.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:43 smithi079 ceph-mon[98036]: osd.4 [v2:172.21.15.79:6808/4037011752,v1:172.21.15.79:6809/4037011752] boot 2023-07-20T17:45:44.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:43 smithi079 ceph-mon[98036]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T17:45:44.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:43 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T17:45:44.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:43 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:45:44.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:43 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:45:44.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:43 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:45.253 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:44 smithi097 ceph-mon[98398]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:45:45.253 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:44 smithi097 ceph-mon[98398]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T17:45:45.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:44 smithi079 ceph-mon[98036]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:45:45.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:44 smithi079 ceph-mon[98036]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T17:45:45.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:44 smithi049 ceph-mon[96530]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:45:45.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:44 smithi049 ceph-mon[96530]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T17:45:46.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:45 smithi097 ceph-mon[98398]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:46.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:45 smithi079 ceph-mon[98036]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:46.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:45 smithi049 ceph-mon[96530]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:47.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:46 smithi097 ceph-mon[98398]: from='client.? 172.21.15.97:0/3178733639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def1f97c-84c4-4c87-9363-62f72142e0a5"}]: dispatch 2023-07-20T17:45:47.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:46 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def1f97c-84c4-4c87-9363-62f72142e0a5"}]: dispatch 2023-07-20T17:45:47.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:46 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "def1f97c-84c4-4c87-9363-62f72142e0a5"}]': finished 2023-07-20T17:45:47.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:46 smithi097 ceph-mon[98398]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T17:45:47.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:45:47.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:46 smithi097 ceph-mon[98398]: from='client.? 172.21.15.97:0/639604485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:45:47.316 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:46 smithi079 ceph-mon[98036]: from='client.? 172.21.15.97:0/3178733639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def1f97c-84c4-4c87-9363-62f72142e0a5"}]: dispatch 2023-07-20T17:45:47.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:46 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def1f97c-84c4-4c87-9363-62f72142e0a5"}]: dispatch 2023-07-20T17:45:47.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:46 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "def1f97c-84c4-4c87-9363-62f72142e0a5"}]': finished 2023-07-20T17:45:47.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:46 smithi079 ceph-mon[98036]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T17:45:47.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:45:47.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:46 smithi079 ceph-mon[98036]: from='client.? 172.21.15.97:0/639604485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:45:47.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:46 smithi049 ceph-mon[96530]: from='client.? 172.21.15.97:0/3178733639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def1f97c-84c4-4c87-9363-62f72142e0a5"}]: dispatch 2023-07-20T17:45:47.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:46 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def1f97c-84c4-4c87-9363-62f72142e0a5"}]: dispatch 2023-07-20T17:45:47.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:46 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "def1f97c-84c4-4c87-9363-62f72142e0a5"}]': finished 2023-07-20T17:45:47.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:46 smithi049 ceph-mon[96530]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T17:45:47.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:45:47.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:46 smithi049 ceph-mon[96530]: from='client.? 172.21.15.97:0/639604485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:45:48.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:48 smithi049 ceph-mon[96530]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:48.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:48 smithi097 ceph-mon[98398]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:48.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:48 smithi079 ceph-mon[98036]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:50.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:50 smithi049 ceph-mon[96530]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:50.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:50 smithi097 ceph-mon[98398]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:50.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:50 smithi079 ceph-mon[98036]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:52.728 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:52 smithi097 ceph-mon[98398]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:52.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:52 smithi049 ceph-mon[96530]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:52.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:52 smithi079 ceph-mon[98036]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:53.673 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T17:45:53.674 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:53.674 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:53 smithi097 ceph-mon[98398]: Deploying daemon osd.5 on smithi097 2023-07-20T17:45:53.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T17:45:53.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:53.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:53 smithi049 ceph-mon[96530]: Deploying daemon osd.5 on smithi097 2023-07-20T17:45:53.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T17:45:53.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:45:53.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:53 smithi079 ceph-mon[98036]: Deploying daemon osd.5 on smithi097 2023-07-20T17:45:54.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:54 smithi097 ceph-mon[98398]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:54.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:54 smithi049 ceph-mon[96530]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:54.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:54 smithi079 ceph-mon[98036]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:56.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:56 smithi049 ceph-mon[96530]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:56.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:56 smithi097 ceph-mon[98398]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:56.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:56 smithi079 ceph-mon[98036]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:58.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:58 smithi049 ceph-mon[96530]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:58.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:58.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:58.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:45:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:58.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:58 smithi097 ceph-mon[98398]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:58.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:58.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:58.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:45:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:45:58.566 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:58 smithi079 ceph-mon[98036]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:45:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:45:58.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:45:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:00.179 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.178+0000 7fcd0e7fc700 1 -- 172.21.15.97:0/1420531591 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcd18060bb0 con 0x7fcd040a11d0 2023-07-20T17:46:00.180 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 5 on host 'smithi097' 2023-07-20T17:46:00.181 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 -- 172.21.15.97:0/1420531591 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fcd040a11d0 msgr2=0x7fcd040a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:00.182 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fcd040a11d0 0x7fcd040a3690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcd14005950 tx=0x7fcd1400a300 comp rx=0 tx=0).stop 2023-07-20T17:46:00.182 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 -- 172.21.15.97:0/1420531591 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd180ff700 msgr2=0x7fcd181953c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:00.182 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd180ff700 0x7fcd181953c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcd08005b40 tx=0x7fcd0800f690 comp rx=0 tx=0).stop 2023-07-20T17:46:00.182 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 -- 172.21.15.97:0/1420531591 shutdown_connections 2023-07-20T17:46:00.183 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fcd040a11d0 0x7fcd040a3690 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:00.183 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd180ff700 0x7fcd181953c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:00.183 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd181012a0 0x7fcd1819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:00.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 --2- 172.21.15.97:0/1420531591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcd18100900 0x7fcd18195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:00.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 -- 172.21.15.97:0/1420531591 >> 172.21.15.97:0/1420531591 conn(0x7fcd180fac80 msgr2=0x7fcd180fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:46:00.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 -- 172.21.15.97:0/1420531591 shutdown_connections 2023-07-20T17:46:00.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:00.180+0000 7fcd1f32d700 1 -- 172.21.15.97:0/1420531591 wait complete. 2023-07-20T17:46:00.586 DEBUG:teuthology.orchestra.run.smithi097:osd.5> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.5.service 2023-07-20T17:46:00.589 INFO:tasks.cephadm:Deploying osd.6 on smithi049 with /dev/vg_nvme/lv_2... 2023-07-20T17:46:00.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T17:46:00.618 INFO:journalctl@ceph.osd.5.smithi097.stdout:-- Logs begin at Thu 2023-07-20 17:15:39 UTC. -- 2023-07-20T17:46:00.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:00 smithi049 ceph-mon[96530]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:46:00.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:00.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:00.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:00.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:00 smithi097 ceph-mon[98398]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:46:00.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:00.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:00.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:00.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:00 smithi079 ceph-mon[98036]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:46:00.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:00.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:00.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:00.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:00.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:01.168 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:46:01.414 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:46:01 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:46:01.129+0000 7f4858515640 -1 osd.5 0 log_to_monitors true 2023-07-20T17:46:01.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:01 smithi049 ceph-mon[96530]: from='osd.5 [v2:172.21.15.97:6808/2172761707,v1:172.21.15.97:6809/2172761707]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T17:46:01.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:01 smithi049 ceph-mon[96530]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T17:46:01.682 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:01 smithi097 ceph-mon[98398]: from='osd.5 [v2:172.21.15.97:6808/2172761707,v1:172.21.15.97:6809/2172761707]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T17:46:01.683 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:01 smithi097 ceph-mon[98398]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T17:46:01.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:01 smithi079 ceph-mon[98036]: from='osd.5 [v2:172.21.15.97:6808/2172761707,v1:172.21.15.97:6809/2172761707]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T17:46:01.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:01 smithi079 ceph-mon[98036]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T17:46:02.639 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T17:46:02.639 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T17:46:02.639 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2023-07-20T17:46:02.639 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2023-07-20T17:46:02.639 INFO:teuthology.orchestra.run.smithi049.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00814435 s, 1.3 GB/s 2023-07-20T17:46:02.640 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 2023-07-20T17:46:02.640 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2023-07-20T17:46:02.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:02 smithi049 ceph-mon[96530]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:46:02.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:02 smithi049 ceph-mon[96530]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T17:46:02.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:02 smithi049 ceph-mon[96530]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T17:46:02.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:02.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:02 smithi049 ceph-mon[96530]: from='osd.5 [v2:172.21.15.97:6808/2172761707,v1:172.21.15.97:6809/2172761707]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:46:02.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:02 smithi049 ceph-mon[96530]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:46:02.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:02 smithi097 ceph-mon[98398]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:46:02.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:02 smithi097 ceph-mon[98398]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T17:46:02.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:02 smithi097 ceph-mon[98398]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T17:46:02.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:02.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:02 smithi097 ceph-mon[98398]: from='osd.5 [v2:172.21.15.97:6808/2172761707,v1:172.21.15.97:6809/2172761707]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:46:02.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:02 smithi097 ceph-mon[98398]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:46:02.768 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:46:02 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:46:02.452+0000 7f484db8d700 -1 osd.5 0 waiting for initial osdmap 2023-07-20T17:46:02.768 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:46:02 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:46:02.461+0000 7f484797f700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:46:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:02 smithi079 ceph-mon[98036]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:46:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:02 smithi079 ceph-mon[98036]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T17:46:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:02 smithi079 ceph-mon[98036]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T17:46:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:02 smithi079 ceph-mon[98036]: from='osd.5 [v2:172.21.15.97:6808/2172761707,v1:172.21.15.97:6809/2172761707]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:46:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:02 smithi079 ceph-mon[98036]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:46:03.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi049:vg_nvme/lv_2 2023-07-20T17:46:03.588 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:46:03.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:46:03.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T17:46:03.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:03.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:03.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:03.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:03.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:03.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:03.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:03.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:03.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:03.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:03 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:03.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:46:03.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T17:46:03.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:03.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:03.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:03.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:03.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:03.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:03.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:03.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:03.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:03.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:03 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:03.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:46:03.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T17:46:03.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:03.821 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:03.821 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:03.822 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:03.822 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:03.822 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:03.822 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:03.822 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:03.822 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:03.822 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:03 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:04.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:04 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:46:04.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:04 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:46:04.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:04 smithi049 ceph-mon[96530]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:46:04.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:04 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi097 2023-07-20T17:46:04.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:04 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi097 to 8584M 2023-07-20T17:46:04.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:04 smithi049 ceph-mon[96530]: osd.5 [v2:172.21.15.97:6808/2172761707,v1:172.21.15.97:6809/2172761707] boot 2023-07-20T17:46:04.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:04 smithi049 ceph-mon[96530]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T17:46:04.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:04 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:04.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:04 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:46:04.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:04 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:46:04.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:04 smithi097 ceph-mon[98398]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:46:04.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:04 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi097 2023-07-20T17:46:04.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:04 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi097 to 8584M 2023-07-20T17:46:04.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:04 smithi097 ceph-mon[98398]: osd.5 [v2:172.21.15.97:6808/2172761707,v1:172.21.15.97:6809/2172761707] boot 2023-07-20T17:46:04.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:04 smithi097 ceph-mon[98398]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T17:46:04.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:04 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:04.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:04 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:46:04.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:04 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:46:04.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:04 smithi079 ceph-mon[98036]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T17:46:04.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:04 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi097 2023-07-20T17:46:04.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:04 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi097 to 8584M 2023-07-20T17:46:04.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:04 smithi079 ceph-mon[98036]: osd.5 [v2:172.21.15.97:6808/2172761707,v1:172.21.15.97:6809/2172761707] boot 2023-07-20T17:46:04.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:04 smithi079 ceph-mon[98036]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T17:46:04.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:04 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T17:46:04.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.861+0000 7f36306fe700 1 -- 172.21.15.49:0/1077559781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36281012a0 msgr2=0x7f3628107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:04.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.861+0000 7f36306fe700 1 --2- 172.21.15.49:0/1077559781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36281012a0 0x7f3628107b50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f362000b0d0 tx=0x7f362000b3e0 comp rx=0 tx=0).stop 2023-07-20T17:46:04.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.861+0000 7f36306fe700 1 -- 172.21.15.49:0/1077559781 shutdown_connections 2023-07-20T17:46:04.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.861+0000 7f36306fe700 1 --2- 172.21.15.49:0/1077559781 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36280ff700 0x7f36280ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:04.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.861+0000 7f36306fe700 1 --2- 172.21.15.49:0/1077559781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36281012a0 0x7f3628107b50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:04.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.861+0000 7f36306fe700 1 --2- 172.21.15.49:0/1077559781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3628100900 0x7f3628100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:04.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.861+0000 7f36306fe700 1 -- 172.21.15.49:0/1077559781 >> 172.21.15.49:0/1077559781 conn(0x7f36280fac80 msgr2=0x7f36280fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:46:04.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.861+0000 7f36306fe700 1 -- 172.21.15.49:0/1077559781 shutdown_connections 2023-07-20T17:46:04.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.861+0000 7f36306fe700 1 -- 172.21.15.49:0/1077559781 wait complete. 2023-07-20T17:46:04.866 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f36306fe700 1 Processor -- start 2023-07-20T17:46:04.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f36306fe700 1 -- start start 2023-07-20T17:46:04.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f36306fe700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36280ff700 0x7f36281954d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:04.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f36306fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3628100900 0x7f3628195a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:04.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f36306fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36281012a0 0x7f362819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:04.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f36306fe700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362819d060 con 0x7f36281012a0 2023-07-20T17:46:04.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f36306fe700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362819d1a0 con 0x7f3628100900 2023-07-20T17:46:04.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f36306fe700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362819d330 con 0x7f36280ff700 2023-07-20T17:46:04.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f362ec9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36281012a0 0x7f362819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:04.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f362e49a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36280ff700 0x7f36281954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:04.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f362ec9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36281012a0 0x7f362819ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35776/0 (socket says 172.21.15.49:35776) 2023-07-20T17:46:04.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f362ec9b700 1 -- 172.21.15.49:0/2138257640 learned_addr learned my addr 172.21.15.49:0/2138257640 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:46:04.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f362dc99700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3628100900 0x7f3628195a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:04.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f362ec9b700 1 -- 172.21.15.49:0/2138257640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36280ff700 msgr2=0x7f36281954d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:04.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f362ec9b700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36280ff700 0x7f36281954d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:04.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f362ec9b700 1 -- 172.21.15.49:0/2138257640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3628100900 msgr2=0x7f3628195a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:04.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f362ec9b700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3628100900 0x7f3628195a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:04.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.862+0000 7f362ec9b700 1 -- 172.21.15.49:0/2138257640 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3620009d00 con 0x7f36281012a0 2023-07-20T17:46:04.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.863+0000 7f362ec9b700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36281012a0 0x7f362819ca90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f362000d010 tx=0x7f36200094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:46:04.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.863+0000 7f361f7fe700 1 -- 172.21.15.49:0/2138257640 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362000e040 con 0x7f36281012a0 2023-07-20T17:46:04.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.863+0000 7f361f7fe700 1 -- 172.21.15.49:0/2138257640 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3620003e10 con 0x7f36281012a0 2023-07-20T17:46:04.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.863+0000 7f36306fe700 1 -- 172.21.15.49:0/2138257640 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362819d5b0 con 0x7f36281012a0 2023-07-20T17:46:04.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.863+0000 7f361f7fe700 1 -- 172.21.15.49:0/2138257640 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620012ab0 con 0x7f36281012a0 2023-07-20T17:46:04.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.863+0000 7f36306fe700 1 -- 172.21.15.49:0/2138257640 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362819d9c0 con 0x7f36281012a0 2023-07-20T17:46:04.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.864+0000 7f36306fe700 1 -- 172.21.15.49:0/2138257640 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362810af30 con 0x7f36281012a0 2023-07-20T17:46:04.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.865+0000 7f361f7fe700 1 -- 172.21.15.49:0/2138257640 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f36200056c0 con 0x7f36281012a0 2023-07-20T17:46:04.871 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.865+0000 7f361f7fe700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f36140a1150 0x7f36140a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:04.871 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.866+0000 7f362e49a700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f36140a1150 0x7f36140a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:04.871 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.866+0000 7f361f7fe700 1 -- 172.21.15.49:0/2138257640 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f36200cd170 con 0x7f36281012a0 2023-07-20T17:46:04.871 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.866+0000 7f362e49a700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f36140a1150 0x7f36140a3610 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3624005950 tx=0x7f362400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:46:04.872 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:04.870+0000 7f361f7fe700 1 -- 172.21.15.49:0/2138257640 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f36200977f0 con 0x7f36281012a0 2023-07-20T17:46:05.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:05.075+0000 7f36306fe700 1 -- 172.21.15.49:0/2138257640 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3628068ac0 con 0x7f36140a1150 2023-07-20T17:46:05.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:05 smithi049 ceph-mon[96530]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T17:46:05.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:46:05.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:46:05.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:05.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:05 smithi097 ceph-mon[98398]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T17:46:05.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:46:05.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:46:05.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:05.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:05 smithi079 ceph-mon[98036]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T17:46:05.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:46:05.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:46:05.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:06.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:06 smithi049 ceph-mon[96530]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:06.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:06 smithi049 ceph-mon[96530]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:46:06.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:06 smithi049 ceph-mon[96530]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T17:46:06.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:06 smithi097 ceph-mon[98398]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:06.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:06 smithi097 ceph-mon[98398]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:46:06.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:06 smithi097 ceph-mon[98398]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T17:46:06.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:06 smithi079 ceph-mon[98036]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:06.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:06 smithi079 ceph-mon[98036]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:46:06.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:06 smithi079 ceph-mon[98036]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T17:46:07.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:07 smithi049 ceph-mon[96530]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T17:46:07.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:07 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2326322096' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8c200c2-0692-4d5e-a281-77c9ad6670cf"}]: dispatch 2023-07-20T17:46:07.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:07 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2326322096' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8c200c2-0692-4d5e-a281-77c9ad6670cf"}]': finished 2023-07-20T17:46:07.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:07 smithi049 ceph-mon[96530]: osdmap e41: 7 total, 6 up, 7 in 2023-07-20T17:46:07.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:07 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:07.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:07 smithi097 ceph-mon[98398]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T17:46:07.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:07 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/2326322096' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8c200c2-0692-4d5e-a281-77c9ad6670cf"}]: dispatch 2023-07-20T17:46:07.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:07 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/2326322096' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8c200c2-0692-4d5e-a281-77c9ad6670cf"}]': finished 2023-07-20T17:46:07.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:07 smithi097 ceph-mon[98398]: osdmap e41: 7 total, 6 up, 7 in 2023-07-20T17:46:07.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:07 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:07.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:07 smithi079 ceph-mon[98036]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T17:46:07.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:07 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/2326322096' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8c200c2-0692-4d5e-a281-77c9ad6670cf"}]: dispatch 2023-07-20T17:46:07.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:07 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/2326322096' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8c200c2-0692-4d5e-a281-77c9ad6670cf"}]': finished 2023-07-20T17:46:07.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:07 smithi079 ceph-mon[98036]: osdmap e41: 7 total, 6 up, 7 in 2023-07-20T17:46:07.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:07 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:08.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:08 smithi049 ceph-mon[96530]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:08.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:08 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/448259209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:46:08.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:08 smithi097 ceph-mon[98398]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:08.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:08 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/448259209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:46:08.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:08 smithi079 ceph-mon[98036]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:08.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:08 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/448259209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:46:10.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:10 smithi049 ceph-mon[96530]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:10.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:10 smithi097 ceph-mon[98398]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:10.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:10 smithi079 ceph-mon[98036]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:12.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:12 smithi049 ceph-mon[96530]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T17:46:12.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:12 smithi097 ceph-mon[98398]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T17:46:12.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:12 smithi079 ceph-mon[98036]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T17:46:14.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:14 smithi049 ceph-mon[96530]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-20T17:46:14.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:14 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T17:46:14.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:14 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:14.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:14 smithi097 ceph-mon[98398]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-20T17:46:14.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:14 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T17:46:14.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:14 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:14.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:14 smithi079 ceph-mon[98036]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-20T17:46:14.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:14 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T17:46:14.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:14 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:15.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:15 smithi049 ceph-mon[96530]: Deploying daemon osd.6 on smithi049 2023-07-20T17:46:15.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:15 smithi097 ceph-mon[98398]: Deploying daemon osd.6 on smithi049 2023-07-20T17:46:15.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:15 smithi079 ceph-mon[98036]: Deploying daemon osd.6 on smithi049 2023-07-20T17:46:16.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:16 smithi049 ceph-mon[96530]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-07-20T17:46:16.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:16 smithi097 ceph-mon[98398]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-07-20T17:46:16.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:16 smithi079 ceph-mon[98036]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-07-20T17:46:18.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:18 smithi049 ceph-mon[96530]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T17:46:18.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:18.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:18.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:18 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:18.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:18 smithi097 ceph-mon[98398]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T17:46:18.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:18.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:18.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:18 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:18.816 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:18 smithi079 ceph-mon[98036]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T17:46:18.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:18.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:18.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:18 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:20.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:20 smithi097 ceph-mon[98398]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-20T17:46:20.769 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:20 smithi049 ceph-mon[96530]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-20T17:46:20.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:20 smithi079 ceph-mon[98036]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-20T17:46:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.029+0000 7f361f7fe700 1 -- 172.21.15.49:0/2138257640 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3628068ac0 con 0x7f36140a1150 2023-07-20T17:46:21.031 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 6 on host 'smithi049' 2023-07-20T17:46:21.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 -- 172.21.15.49:0/2138257640 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f36140a1150 msgr2=0x7f36140a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:21.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f36140a1150 0x7f36140a3610 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3624005950 tx=0x7f362400a300 comp rx=0 tx=0).stop 2023-07-20T17:46:21.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 -- 172.21.15.49:0/2138257640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36281012a0 msgr2=0x7f362819ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:21.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36281012a0 0x7f362819ca90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f362000d010 tx=0x7f36200094c0 comp rx=0 tx=0).stop 2023-07-20T17:46:21.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 -- 172.21.15.49:0/2138257640 shutdown_connections 2023-07-20T17:46:21.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f36140a1150 0x7f36140a3610 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:21.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36280ff700 0x7f36281954d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:21.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36281012a0 0x7f362819ca90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:21.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 --2- 172.21.15.49:0/2138257640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3628100900 0x7f3628195a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:21.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 -- 172.21.15.49:0/2138257640 >> 172.21.15.49:0/2138257640 conn(0x7f36280fac80 msgr2=0x7f3628103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:46:21.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 -- 172.21.15.49:0/2138257640 shutdown_connections 2023-07-20T17:46:21.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:46:21.031+0000 7f36306fe700 1 -- 172.21.15.49:0/2138257640 wait complete. 2023-07-20T17:46:21.373 DEBUG:teuthology.orchestra.run.smithi049:osd.6> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.6.service 2023-07-20T17:46:21.376 INFO:tasks.cephadm:Deploying osd.7 on smithi079 with /dev/vg_nvme/lv_2... 2023-07-20T17:46:21.376 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T17:46:21.406 INFO:journalctl@ceph.osd.6.smithi049.stdout:-- Logs begin at Thu 2023-07-20 17:16:49 UTC. -- 2023-07-20T17:46:21.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:21.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:21.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:21.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:21.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:21.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:21.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:21.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:21.899 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:21.899 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:21.899 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:21.900 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:21.900 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:21.900 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:21.900 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:21.900 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:21.949 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T17:46:22.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:22.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:22.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:22.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:22.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:22.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:22.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:22.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:22.568 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:46:22 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:46:22.277+0000 7f2f49820640 -1 osd.6 0 log_to_monitors true 2023-07-20T17:46:22.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:22 smithi079 ceph-mon[98036]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T17:46:22.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:22 smithi079 ceph-mon[98036]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T17:46:22.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:22 smithi049 ceph-mon[96530]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T17:46:22.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:22 smithi049 ceph-mon[96530]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T17:46:23.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:22 smithi097 ceph-mon[98398]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T17:46:23.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:22 smithi097 ceph-mon[98398]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T17:46:23.515 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T17:46:23.515 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T17:46:23.516 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2023-07-20T17:46:23.516 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2023-07-20T17:46:23.516 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111157 s, 943 MB/s 2023-07-20T17:46:23.516 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2023-07-20T17:46:23.739 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:46:23.653+0000 7f2f3ee98700 -1 osd.6 0 waiting for initial osdmap 2023-07-20T17:46:23.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:46:23.661+0000 7f2f3948b700 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:46:23.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T17:46:23.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T17:46:23.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:23.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:46:23.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:23.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:23.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:23.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:23.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:23.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:23.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:23.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:23.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:23.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T17:46:23.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T17:46:23.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:23.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:46:23.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:23.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:23.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:23.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:23.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:23.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:23.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:23.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:23.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:23.887 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi079:vg_nvme/lv_2 2023-07-20T17:46:24.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T17:46:24.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T17:46:24.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:24.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:46:24.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:24.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:24.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:24.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:24.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:24.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:24.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:24.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:24.021 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:24.439 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T17:46:24.894 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:24 smithi079 ceph-mon[98036]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:24.894 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:24 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi049 2023-07-20T17:46:24.895 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:24 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi049 to 5722M 2023-07-20T17:46:24.895 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:24 smithi079 ceph-mon[98036]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:46:24.895 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:24 smithi079 ceph-mon[98036]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T17:46:24.895 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:24 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:24.895 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:24 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:24.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:24 smithi049 ceph-mon[96530]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:24.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:24 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi049 2023-07-20T17:46:24.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:24 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi049 to 5722M 2023-07-20T17:46:24.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:24 smithi049 ceph-mon[96530]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:46:24.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:24 smithi049 ceph-mon[96530]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T17:46:24.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:24 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:24.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:24 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:25.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:24 smithi097 ceph-mon[98398]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T17:46:25.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:24 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi049 2023-07-20T17:46:25.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:24 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi049 to 5722M 2023-07-20T17:46:25.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:24 smithi097 ceph-mon[98398]: from='osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:46:25.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:24 smithi097 ceph-mon[98398]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T17:46:25.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:24 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:25.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:24 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:25.531 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.529+0000 7fab484bb700 1 -- 172.21.15.79:0/68830888 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab40100900 msgr2=0x7fab40100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:25.532 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.529+0000 7fab484bb700 1 --2- 172.21.15.79:0/68830888 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab40100900 0x7fab40100d60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fab300099c0 tx=0x7fab30009cd0 comp rx=0 tx=0).stop 2023-07-20T17:46:25.532 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.529+0000 7fab484bb700 1 -- 172.21.15.79:0/68830888 shutdown_connections 2023-07-20T17:46:25.532 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.529+0000 7fab484bb700 1 --2- 172.21.15.79:0/68830888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab400ff700 0x7fab400ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:25.532 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.529+0000 7fab484bb700 1 --2- 172.21.15.79:0/68830888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab401012a0 0x7fab40107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:25.532 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.529+0000 7fab484bb700 1 --2- 172.21.15.79:0/68830888 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab40100900 0x7fab40100d60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:25.532 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.529+0000 7fab484bb700 1 -- 172.21.15.79:0/68830888 >> 172.21.15.79:0/68830888 conn(0x7fab400fac80 msgr2=0x7fab400fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:46:25.533 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.529+0000 7fab484bb700 1 -- 172.21.15.79:0/68830888 shutdown_connections 2023-07-20T17:46:25.533 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.529+0000 7fab484bb700 1 -- 172.21.15.79:0/68830888 wait complete. 2023-07-20T17:46:25.533 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.529+0000 7fab484bb700 1 Processor -- start 2023-07-20T17:46:25.533 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.529+0000 7fab484bb700 1 -- start start 2023-07-20T17:46:25.533 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab484bb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab400ff700 0x7fab40195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:25.533 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab484bb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab40100900 0x7fab401959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:25.533 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab484bb700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab401012a0 0x7fab4019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:25.533 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab484bb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4019d020 con 0x7fab40100900 2023-07-20T17:46:25.534 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab484bb700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4019d160 con 0x7fab400ff700 2023-07-20T17:46:25.534 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab484bb700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4019d2f0 con 0x7fab401012a0 2023-07-20T17:46:25.534 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab46257700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab400ff700 0x7fab40195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:25.534 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab46257700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab400ff700 0x7fab40195490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47934/0 (socket says 172.21.15.79:47934) 2023-07-20T17:46:25.534 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab46257700 1 -- 172.21.15.79:0/1829190088 learned_addr learned my addr 172.21.15.79:0/1829190088 (peer_addr_for_me v2:172.21.15.79:0/0) 2023-07-20T17:46:25.534 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab46a58700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab401012a0 0x7fab4019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:25.534 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab45a56700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab40100900 0x7fab401959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:25.534 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab46257700 1 -- 172.21.15.79:0/1829190088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab401012a0 msgr2=0x7fab4019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:25.535 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab46257700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab401012a0 0x7fab4019ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:25.535 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab46257700 1 -- 172.21.15.79:0/1829190088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab40100900 msgr2=0x7fab401959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:25.535 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab46257700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab40100900 0x7fab401959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:25.535 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab46257700 1 -- 172.21.15.79:0/1829190088 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab300095f0 con 0x7fab400ff700 2023-07-20T17:46:25.535 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab46a58700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab401012a0 0x7fab4019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:46:25.535 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab46257700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab400ff700 0x7fab40195490 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00b500 tx=0x7fab3c00b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:46:25.535 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab377fe700 1 -- 172.21.15.79:0/1829190088 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c011830 con 0x7fab400ff700 2023-07-20T17:46:25.536 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab377fe700 1 -- 172.21.15.79:0/1829190088 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab3c011e50 con 0x7fab400ff700 2023-07-20T17:46:25.536 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab484bb700 1 -- 172.21.15.79:0/1829190088 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab400683d0 con 0x7fab400ff700 2023-07-20T17:46:25.536 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab377fe700 1 -- 172.21.15.79:0/1829190088 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c00f2e0 con 0x7fab400ff700 2023-07-20T17:46:25.536 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.530+0000 7fab484bb700 1 -- 172.21.15.79:0/1829190088 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab400689a0 con 0x7fab400ff700 2023-07-20T17:46:25.537 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.532+0000 7fab484bb700 1 -- 172.21.15.79:0/1829190088 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab40106c90 con 0x7fab400ff700 2023-07-20T17:46:25.537 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.533+0000 7fab377fe700 1 -- 172.21.15.79:0/1829190088 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fab3c010580 con 0x7fab400ff700 2023-07-20T17:46:25.537 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.533+0000 7fab377fe700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab2c0a11d0 0x7fab2c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:25.537 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.533+0000 7fab377fe700 1 -- 172.21.15.79:0/1829190088 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fab3c0cb640 con 0x7fab400ff700 2023-07-20T17:46:25.537 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.533+0000 7fab45a56700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab2c0a11d0 0x7fab2c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:25.538 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.534+0000 7fab45a56700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab2c0a11d0 0x7fab2c0a3690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fab300098e0 tx=0x7fab3000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:46:25.538 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.537+0000 7fab377fe700 1 -- 172.21.15.79:0/1829190088 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fab3c095aa0 con 0x7fab400ff700 2023-07-20T17:46:25.733 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:25.732+0000 7fab484bb700 1 -- 172.21.15.79:0/1829190088 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fab40068c20 con 0x7fab2c0a11d0 2023-07-20T17:46:25.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:25 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:46:25.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:25 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:46:25.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:25 smithi049 ceph-mon[96530]: osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207] boot 2023-07-20T17:46:25.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:25 smithi049 ceph-mon[96530]: osdmap e44: 7 total, 7 up, 7 in 2023-07-20T17:46:25.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:26.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:25 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:46:26.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:25 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:46:26.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:25 smithi097 ceph-mon[98398]: osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207] boot 2023-07-20T17:46:26.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:25 smithi097 ceph-mon[98398]: osdmap e44: 7 total, 7 up, 7 in 2023-07-20T17:46:26.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:25 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:26.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:25 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:46:26.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:25 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:46:26.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:25 smithi079 ceph-mon[98036]: osd.6 [v2:172.21.15.49:6818/797242207,v1:172.21.15.49:6819/797242207] boot 2023-07-20T17:46:26.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:25 smithi079 ceph-mon[98036]: osdmap e44: 7 total, 7 up, 7 in 2023-07-20T17:46:26.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:25 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T17:46:26.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:26 smithi079 ceph-mon[98036]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:26.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:26 smithi079 ceph-mon[98036]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T17:46:26.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:26 smithi079 ceph-mon[98036]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:46:26.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:46:26.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:46:26.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:26.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:26 smithi049 ceph-mon[96530]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:26.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:26 smithi049 ceph-mon[96530]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T17:46:26.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:26 smithi049 ceph-mon[96530]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:46:26.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:46:26.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:46:26.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:27.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:26 smithi097 ceph-mon[98398]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:27.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:26 smithi097 ceph-mon[98398]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T17:46:27.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:26 smithi097 ceph-mon[98398]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:46:27.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:46:27.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:46:27.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:28.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:28 smithi049 ceph-mon[96530]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:28.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:28 smithi049 ceph-mon[96530]: from='client.? 172.21.15.79:0/243987208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f"}]: dispatch 2023-07-20T17:46:28.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:28 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f"}]: dispatch 2023-07-20T17:46:28.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:28 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f"}]': finished 2023-07-20T17:46:28.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:28 smithi049 ceph-mon[96530]: osdmap e46: 8 total, 7 up, 8 in 2023-07-20T17:46:28.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:29.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:28 smithi097 ceph-mon[98398]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:29.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:28 smithi097 ceph-mon[98398]: from='client.? 172.21.15.79:0/243987208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f"}]: dispatch 2023-07-20T17:46:29.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:28 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f"}]: dispatch 2023-07-20T17:46:29.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:28 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f"}]': finished 2023-07-20T17:46:29.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:28 smithi097 ceph-mon[98398]: osdmap e46: 8 total, 7 up, 8 in 2023-07-20T17:46:29.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:29.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:28 smithi079 ceph-mon[98036]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:29.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:28 smithi079 ceph-mon[98036]: from='client.? 172.21.15.79:0/243987208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f"}]: dispatch 2023-07-20T17:46:29.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:28 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f"}]: dispatch 2023-07-20T17:46:29.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:28 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f"}]': finished 2023-07-20T17:46:29.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:28 smithi079 ceph-mon[98036]: osdmap e46: 8 total, 7 up, 8 in 2023-07-20T17:46:29.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:29.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:29 smithi049 ceph-mon[96530]: from='client.? 172.21.15.79:0/2251863632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:46:30.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:29 smithi097 ceph-mon[98398]: from='client.? 172.21.15.79:0/2251863632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:46:30.066 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:29 smithi079 ceph-mon[98036]: from='client.? 172.21.15.79:0/2251863632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:46:30.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:30 smithi049 ceph-mon[96530]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:31.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:30 smithi097 ceph-mon[98398]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:31.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:30 smithi079 ceph-mon[98036]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:32.935 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:32 smithi079 ceph-mon[98036]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:32.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:32 smithi049 ceph-mon[96530]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:33.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:32 smithi097 ceph-mon[98398]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:34.936 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:34 smithi079 ceph-mon[98036]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:34.937 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T17:46:34.937 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:34.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:34 smithi049 ceph-mon[96530]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:34.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T17:46:34.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:35.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:34 smithi097 ceph-mon[98398]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:35.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T17:46:35.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:35.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:35 smithi049 ceph-mon[96530]: Deploying daemon osd.7 on smithi079 2023-07-20T17:46:36.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:35 smithi097 ceph-mon[98398]: Deploying daemon osd.7 on smithi079 2023-07-20T17:46:36.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:35 smithi079 ceph-mon[98036]: Deploying daemon osd.7 on smithi079 2023-07-20T17:46:36.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:36 smithi049 ceph-mon[96530]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:37.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:36 smithi097 ceph-mon[98398]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:37.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:36 smithi079 ceph-mon[98036]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:38.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:38 smithi049 ceph-mon[96530]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:38.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:38.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:38.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:39.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:38 smithi097 ceph-mon[98398]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:39.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:39.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:39.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:39.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:38 smithi079 ceph-mon[98036]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:39.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:39.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:39.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:40.984 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:40 smithi079 ceph-mon[98036]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:40.984 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:40 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:40.984 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:40 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:40.984 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:40 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:40.984 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:40 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:40.985 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:40 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:40.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:40 smithi049 ceph-mon[96530]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:40.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:40 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:40.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:40 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:40.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:40 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:40.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:40 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:40.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:40 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:41.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:40 smithi097 ceph-mon[98398]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:41.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:40 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:41.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:40 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:41.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:40 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:41.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:40 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:41.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:40 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:41.047 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.044+0000 7fab377fe700 1 -- 172.21.15.79:0/1829190088 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fab40068c20 con 0x7fab2c0a11d0 2023-07-20T17:46:41.047 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 7 on host 'smithi079' 2023-07-20T17:46:41.048 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 -- 172.21.15.79:0/1829190088 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab2c0a11d0 msgr2=0x7fab2c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:41.048 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab2c0a11d0 0x7fab2c0a3690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fab300098e0 tx=0x7fab3000b540 comp rx=0 tx=0).stop 2023-07-20T17:46:41.048 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 -- 172.21.15.79:0/1829190088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab400ff700 msgr2=0x7fab40195490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:41.049 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab400ff700 0x7fab40195490 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00b500 tx=0x7fab3c00b8c0 comp rx=0 tx=0).stop 2023-07-20T17:46:41.049 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 -- 172.21.15.79:0/1829190088 shutdown_connections 2023-07-20T17:46:41.049 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab2c0a11d0 0x7fab2c0a3690 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:41.049 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab401012a0 0x7fab4019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:41.049 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab400ff700 0x7fab40195490 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:41.049 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 --2- 172.21.15.79:0/1829190088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab40100900 0x7fab401959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:41.049 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 -- 172.21.15.79:0/1829190088 >> 172.21.15.79:0/1829190088 conn(0x7fab400fac80 msgr2=0x7fab40103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:46:41.050 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 -- 172.21.15.79:0/1829190088 shutdown_connections 2023-07-20T17:46:41.050 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:46:41.046+0000 7fab484bb700 1 -- 172.21.15.79:0/1829190088 wait complete. 2023-07-20T17:46:41.431 DEBUG:teuthology.orchestra.run.smithi079:osd.7> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.7.service 2023-07-20T17:46:41.433 INFO:tasks.cephadm:Deploying osd.8 on smithi097 with /dev/vg_nvme/lv_2... 2023-07-20T17:46:41.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T17:46:41.463 INFO:journalctl@ceph.osd.7.smithi079.stdout:-- Logs begin at Thu 2023-07-20 17:15:42 UTC. -- 2023-07-20T17:46:41.996 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:46:42.165 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:42 smithi079 ceph-mon[98036]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:42.165 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:42.165 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:42.166 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:42.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:42 smithi097 ceph-mon[98398]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:42.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:42.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:42.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:42.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:42 smithi049 ceph-mon[96530]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:42.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:42.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:42.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:46:42.817 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:46:42 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:46:42.489+0000 7f38c79cf640 -1 osd.7 0 log_to_monitors true 2023-07-20T17:46:43.530 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T17:46:43.531 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T17:46:43.531 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2023-07-20T17:46:43.531 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2023-07-20T17:46:43.531 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123947 s, 846 MB/s 2023-07-20T17:46:43.531 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2023-07-20T17:46:43.543 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:43 smithi079 ceph-mon[98036]: from='osd.7 [v2:172.21.15.79:6816/1417643292,v1:172.21.15.79:6817/1417643292]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T17:46:43.543 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:43 smithi079 ceph-mon[98036]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T17:46:43.659 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:43 smithi097 ceph-mon[98398]: from='osd.7 [v2:172.21.15.79:6816/1417643292,v1:172.21.15.79:6817/1417643292]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T17:46:43.659 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:43 smithi097 ceph-mon[98398]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T17:46:43.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:43 smithi049 ceph-mon[96530]: from='osd.7 [v2:172.21.15.79:6816/1417643292,v1:172.21.15.79:6817/1417643292]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T17:46:43.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:43 smithi049 ceph-mon[96530]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T17:46:43.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi097:vg_nvme/lv_2 2023-07-20T17:46:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:46:44.723 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:44.723 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T17:46:44.723 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T17:46:44.723 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:44.724 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='osd.7 [v2:172.21.15.79:6816/1417643292,v1:172.21.15.79:6817/1417643292]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:46:44.724 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:46:44.724 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi079 2023-07-20T17:46:44.724 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:44.724 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:44.724 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:44.724 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:44.725 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:44.725 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi079 to 5722M 2023-07-20T17:46:44.725 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:44.725 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:44.725 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:44.725 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:44.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:44.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T17:46:44.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T17:46:44.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:44.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='osd.7 [v2:172.21.15.79:6816/1417643292,v1:172.21.15.79:6817/1417643292]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:46:44.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:46:44.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi079 2023-07-20T17:46:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi079 to 5722M 2023-07-20T17:46:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:44.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:44.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:44.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:44.817 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:46:44.392+0000 7f38bd047700 -1 osd.7 0 waiting for initial osdmap 2023-07-20T17:46:44.818 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:46:44.401+0000 7f38b5e37700 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:46:44.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:44.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T17:46:44.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T17:46:44.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:44.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='osd.7 [v2:172.21.15.79:6816/1417643292,v1:172.21.15.79:6817/1417643292]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:46:44.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:46:44.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi079 2023-07-20T17:46:44.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:44.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:44.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:44.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:44.821 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:46:44.821 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi079 to 5722M 2023-07-20T17:46:44.821 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:44.821 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:44.822 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:46:44.822 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:46:45.682 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:45 smithi097 ceph-mon[98398]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:46:45.682 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:45 smithi097 ceph-mon[98398]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T17:46:45.682 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:45 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:45.682 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:45 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:45.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:45 smithi049 ceph-mon[96530]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:46:45.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:45 smithi049 ceph-mon[96530]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T17:46:45.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:45 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:45.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:45 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:45.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:45 smithi079 ceph-mon[98036]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:46:45.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:45 smithi079 ceph-mon[98036]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T17:46:45.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:45 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:45.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:45 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:45.824 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.823+0000 7f5570607700 1 -- 172.21.15.97:0/2527404074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568100840 msgr2=0x7f5568100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:45.824 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.823+0000 7f5570607700 1 --2- 172.21.15.97:0/2527404074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568100840 0x7f5568100c60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5558009990 tx=0x7f5558009ca0 comp rx=0 tx=0).stop 2023-07-20T17:46:45.825 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.823+0000 7f5570607700 1 -- 172.21.15.97:0/2527404074 shutdown_connections 2023-07-20T17:46:45.825 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.823+0000 7f5570607700 1 --2- 172.21.15.97:0/2527404074 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568100840 0x7f5568100c60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:45.825 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.823+0000 7f5570607700 1 --2- 172.21.15.97:0/2527404074 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55681023e0 0x7f5568108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:45.825 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.823+0000 7f5570607700 1 --2- 172.21.15.97:0/2527404074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5568101a40 0x7f5568101ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:45.825 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.823+0000 7f5570607700 1 -- 172.21.15.97:0/2527404074 >> 172.21.15.97:0/2527404074 conn(0x7f55680fbde0 msgr2=0x7f55680fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:46:45.825 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.823+0000 7f5570607700 1 -- 172.21.15.97:0/2527404074 shutdown_connections 2023-07-20T17:46:45.825 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.823+0000 7f5570607700 1 -- 172.21.15.97:0/2527404074 wait complete. 2023-07-20T17:46:45.827 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f5570607700 1 Processor -- start 2023-07-20T17:46:45.828 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f5570607700 1 -- start start 2023-07-20T17:46:45.828 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f5570607700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5568100840 0x7f55681965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:45.828 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f5570607700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568101a40 0x7f5568196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:45.828 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f5570607700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55681023e0 0x7f556819db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:45.828 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f5570607700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556819e160 con 0x7f5568100840 2023-07-20T17:46:45.828 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f5570607700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556819e2a0 con 0x7f55681023e0 2023-07-20T17:46:45.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f5570607700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556819e430 con 0x7f5568101a40 2023-07-20T17:46:45.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f556dba2700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568101a40 0x7f5568196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:45.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f556dba2700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568101a40 0x7f5568196b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60848/0 (socket says 172.21.15.97:60848) 2023-07-20T17:46:45.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f556dba2700 1 -- 172.21.15.97:0/2304969556 learned_addr learned my addr 172.21.15.97:0/2304969556 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-07-20T17:46:45.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f556e3a3700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5568100840 0x7f55681965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:45.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f556eba4700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55681023e0 0x7f556819db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:45.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f556dba2700 1 -- 172.21.15.97:0/2304969556 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55681023e0 msgr2=0x7f556819db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:45.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f556dba2700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55681023e0 0x7f556819db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:45.830 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f556dba2700 1 -- 172.21.15.97:0/2304969556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5568100840 msgr2=0x7f55681965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:46:45.830 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f556dba2700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5568100840 0x7f55681965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:46:45.830 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.824+0000 7f556dba2700 1 -- 172.21.15.97:0/2304969556 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55580095f0 con 0x7f5568101a40 2023-07-20T17:46:45.830 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.825+0000 7f556dba2700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568101a40 0x7f5568196b10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f556400b560 tx=0x7f556400b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:46:45.830 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.825+0000 7f555f7fe700 1 -- 172.21.15.97:0/2304969556 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5564011830 con 0x7f5568101a40 2023-07-20T17:46:45.830 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.825+0000 7f5570607700 1 -- 172.21.15.97:0/2304969556 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556819e710 con 0x7f5568101a40 2023-07-20T17:46:45.830 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.825+0000 7f555f7fe700 1 -- 172.21.15.97:0/2304969556 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5564011e50 con 0x7f5568101a40 2023-07-20T17:46:45.831 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.825+0000 7f555f7fe700 1 -- 172.21.15.97:0/2304969556 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556400f2e0 con 0x7f5568101a40 2023-07-20T17:46:45.831 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.825+0000 7f5570607700 1 -- 172.21.15.97:0/2304969556 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556819ec90 con 0x7f5568101a40 2023-07-20T17:46:45.831 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.826+0000 7f5570607700 1 -- 172.21.15.97:0/2304969556 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556810c110 con 0x7f5568101a40 2023-07-20T17:46:45.831 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.827+0000 7f555f7fe700 1 -- 172.21.15.97:0/2304969556 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5564010580 con 0x7f5568101a40 2023-07-20T17:46:45.832 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.827+0000 7f555f7fe700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f55540a11d0 0x7f55540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:46:45.832 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.828+0000 7f555f7fe700 1 -- 172.21.15.97:0/2304969556 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5430+0+0 (secure 0 0 0) 0x7f55640cbe80 con 0x7f5568101a40 2023-07-20T17:46:45.832 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.828+0000 7f556e3a3700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f55540a11d0 0x7f55540a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:46:45.832 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.828+0000 7f556e3a3700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f55540a11d0 0x7f55540a3690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5558000c00 tx=0x7f5558005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:46:45.832 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:45.831+0000 7f555f7fe700 1 -- 172.21.15.97:0/2304969556 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5564096070 con 0x7f5568101a40 2023-07-20T17:46:46.023 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:46:46.021+0000 7f5570607700 1 -- 172.21.15.97:0/2304969556 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5568060bb0 con 0x7f55540a11d0 2023-07-20T17:46:46.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:46 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:46:46.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:46 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:46:46.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:46 smithi049 ceph-mon[96530]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:46.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:46 smithi049 ceph-mon[96530]: osd.7 [v2:172.21.15.79:6816/1417643292,v1:172.21.15.79:6817/1417643292] boot 2023-07-20T17:46:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:46 smithi049 ceph-mon[96530]: osdmap e49: 8 total, 8 up, 8 in 2023-07-20T17:46:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:46:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:46:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:46.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:46 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:46:46.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:46 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:46:46.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:46 smithi097 ceph-mon[98398]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:46.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:46.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:46 smithi097 ceph-mon[98398]: osd.7 [v2:172.21.15.79:6816/1417643292,v1:172.21.15.79:6817/1417643292] boot 2023-07-20T17:46:46.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:46 smithi097 ceph-mon[98398]: osdmap e49: 8 total, 8 up, 8 in 2023-07-20T17:46:46.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:46.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:46:46.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:46:46.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:46 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:46:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:46 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:46:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:46 smithi079 ceph-mon[98036]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T17:46:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:46 smithi079 ceph-mon[98036]: osd.7 [v2:172.21.15.79:6816/1417643292,v1:172.21.15.79:6817/1417643292] boot 2023-07-20T17:46:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:46 smithi079 ceph-mon[98036]: osdmap e49: 8 total, 8 up, 8 in 2023-07-20T17:46:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T17:46:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:46:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:46:46.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:47.658 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:47 smithi097 ceph-mon[98398]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:46:47.658 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:47 smithi097 ceph-mon[98398]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T17:46:47.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:47 smithi049 ceph-mon[96530]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:46:47.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:47 smithi049 ceph-mon[96530]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T17:46:47.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:47 smithi079 ceph-mon[98036]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:46:47.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:47 smithi079 ceph-mon[98036]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T17:46:48.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:48 smithi049 ceph-mon[96530]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:48.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:48 smithi049 ceph-mon[96530]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T17:46:48.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:48 smithi097 ceph-mon[98398]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:48.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:48 smithi097 ceph-mon[98398]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T17:46:48.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:48 smithi079 ceph-mon[98036]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:48.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:48 smithi079 ceph-mon[98036]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T17:46:49.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:49 smithi049 ceph-mon[96530]: from='client.? 172.21.15.97:0/1477283083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c55ac6-0e96-42a4-b974-178c87c2185c"}]: dispatch 2023-07-20T17:46:49.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:49 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c55ac6-0e96-42a4-b974-178c87c2185c"}]: dispatch 2023-07-20T17:46:49.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:49 smithi049 ceph-mon[96530]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T17:46:49.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:49 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14c55ac6-0e96-42a4-b974-178c87c2185c"}]': finished 2023-07-20T17:46:49.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:49 smithi049 ceph-mon[96530]: osdmap e53: 9 total, 8 up, 9 in 2023-07-20T17:46:49.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:49 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:46:49.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:49 smithi049 ceph-mon[96530]: from='client.? 172.21.15.97:0/4190245620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:46:49.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:49 smithi097 ceph-mon[98398]: from='client.? 172.21.15.97:0/1477283083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c55ac6-0e96-42a4-b974-178c87c2185c"}]: dispatch 2023-07-20T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:49 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c55ac6-0e96-42a4-b974-178c87c2185c"}]: dispatch 2023-07-20T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:49 smithi097 ceph-mon[98398]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:49 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14c55ac6-0e96-42a4-b974-178c87c2185c"}]': finished 2023-07-20T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:49 smithi097 ceph-mon[98398]: osdmap e53: 9 total, 8 up, 9 in 2023-07-20T17:46:49.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:49 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:46:49.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:49 smithi097 ceph-mon[98398]: from='client.? 172.21.15.97:0/4190245620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:46:49.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:49 smithi079 ceph-mon[98036]: from='client.? 172.21.15.97:0/1477283083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c55ac6-0e96-42a4-b974-178c87c2185c"}]: dispatch 2023-07-20T17:46:49.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:49 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c55ac6-0e96-42a4-b974-178c87c2185c"}]: dispatch 2023-07-20T17:46:49.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:49 smithi079 ceph-mon[98036]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T17:46:49.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:49 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14c55ac6-0e96-42a4-b974-178c87c2185c"}]': finished 2023-07-20T17:46:49.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:49 smithi079 ceph-mon[98036]: osdmap e53: 9 total, 8 up, 9 in 2023-07-20T17:46:49.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:49 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:46:49.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:49 smithi079 ceph-mon[98036]: from='client.? 172.21.15.97:0/4190245620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:46:50.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:50 smithi049 ceph-mon[96530]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:50.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:50 smithi049 ceph-mon[96530]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T17:46:50.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:50 smithi097 ceph-mon[98398]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:50.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:50 smithi097 ceph-mon[98398]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T17:46:50.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:46:50.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:50 smithi079 ceph-mon[98036]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:50.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:50 smithi079 ceph-mon[98036]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T17:46:50.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:46:52.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:52 smithi049 ceph-mon[96530]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:52.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:52 smithi097 ceph-mon[98398]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:52.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:52 smithi079 ceph-mon[98036]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:54.674 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:54 smithi097 ceph-mon[98398]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:54.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:54 smithi049 ceph-mon[96530]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:54.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:54 smithi079 ceph-mon[98036]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:55.739 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:55 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T17:46:55.740 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:55 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:55.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:55 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T17:46:55.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:55 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:55.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:55 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T17:46:55.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:55 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:46:56.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:56 smithi049 ceph-mon[96530]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:56.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:56 smithi049 ceph-mon[96530]: Deploying daemon osd.8 on smithi097 2023-07-20T17:46:56.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:56 smithi097 ceph-mon[98398]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:56.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:56 smithi097 ceph-mon[98398]: Deploying daemon osd.8 on smithi097 2023-07-20T17:46:56.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:56 smithi079 ceph-mon[98036]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:56.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:56 smithi079 ceph-mon[98036]: Deploying daemon osd.8 on smithi097 2023-07-20T17:46:58.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:46:58 smithi049 ceph-mon[96530]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:58.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:46:58 smithi097 ceph-mon[98398]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:46:58.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:46:58 smithi079 ceph-mon[98036]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:00.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:00 smithi049 ceph-mon[96530]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:00.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:00.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:00.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:00 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:00.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:00 smithi097 ceph-mon[98398]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:00.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:00.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:00.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:00 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:00.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:00 smithi079 ceph-mon[98036]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:00.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:00.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:00.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:00 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.336 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.333+0000 7f555f7fe700 1 -- 172.21.15.97:0/2304969556 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5568060bb0 con 0x7f55540a11d0 2023-07-20T17:47:02.336 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 8 on host 'smithi097' 2023-07-20T17:47:02.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 -- 172.21.15.97:0/2304969556 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f55540a11d0 msgr2=0x7f55540a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:02.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f55540a11d0 0x7f55540a3690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5558000c00 tx=0x7f5558005fb0 comp rx=0 tx=0).stop 2023-07-20T17:47:02.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 -- 172.21.15.97:0/2304969556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568101a40 msgr2=0x7f5568196b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:02.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568101a40 0x7f5568196b10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f556400b560 tx=0x7f556400b870 comp rx=0 tx=0).stop 2023-07-20T17:47:02.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 -- 172.21.15.97:0/2304969556 shutdown_connections 2023-07-20T17:47:02.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f55540a11d0 0x7f55540a3690 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:02.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f55681023e0 0x7f556819db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:02.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5568100840 0x7f55681965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:02.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 --2- 172.21.15.97:0/2304969556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5568101a40 0x7f5568196b10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:02.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 -- 172.21.15.97:0/2304969556 >> 172.21.15.97:0/2304969556 conn(0x7f55680fbde0 msgr2=0x7f55680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:47:02.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 -- 172.21.15.97:0/2304969556 shutdown_connections 2023-07-20T17:47:02.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:02.335+0000 7f5570607700 1 -- 172.21.15.97:0/2304969556 wait complete. 2023-07-20T17:47:02.517 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:02 smithi097 ceph-mon[98398]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:02.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:02.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:02.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:02.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:02 smithi049 ceph-mon[96530]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:02.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:02.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:02.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:02.787 DEBUG:teuthology.orchestra.run.smithi097:osd.8> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.8.service 2023-07-20T17:47:02.789 INFO:tasks.cephadm:Deploying osd.9 on smithi049 with /dev/vg_nvme/lv_1... 2023-07-20T17:47:02.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T17:47:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:02 smithi079 ceph-mon[98036]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:02.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:02.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:02.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:02.819 INFO:journalctl@ceph.osd.8.smithi097.stdout:-- Logs begin at Thu 2023-07-20 17:15:39 UTC. -- 2023-07-20T17:47:03.349 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:47:03.599 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:03 smithi097 ceph-mon[98398]: from='osd.8 [v2:172.21.15.97:6816/1465924719,v1:172.21.15.97:6817/1465924719]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T17:47:03.599 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:03 smithi097 ceph-mon[98398]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T17:47:03.599 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:47:03 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:47:03.425+0000 7f2615702640 -1 osd.8 0 log_to_monitors true 2023-07-20T17:47:03.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:03 smithi079 ceph-mon[98036]: from='osd.8 [v2:172.21.15.97:6816/1465924719,v1:172.21.15.97:6817/1465924719]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T17:47:03.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:03 smithi079 ceph-mon[98036]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T17:47:03.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:03 smithi049 ceph-mon[96530]: from='osd.8 [v2:172.21.15.97:6816/1465924719,v1:172.21.15.97:6817/1465924719]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T17:47:03.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:03 smithi049 ceph-mon[96530]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T17:47:04.507 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:04 smithi097 ceph-mon[98398]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:04.508 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:04 smithi097 ceph-mon[98398]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T17:47:04.508 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:04 smithi097 ceph-mon[98398]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T17:47:04.508 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:04 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:04.508 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:04 smithi097 ceph-mon[98398]: from='osd.8 [v2:172.21.15.97:6816/1465924719,v1:172.21.15.97:6817/1465924719]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:47:04.508 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:04 smithi097 ceph-mon[98398]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:47:04.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:04 smithi049 ceph-mon[96530]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:04.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:04 smithi049 ceph-mon[96530]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T17:47:04.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:04 smithi049 ceph-mon[96530]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T17:47:04.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:04 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:04.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:04 smithi049 ceph-mon[96530]: from='osd.8 [v2:172.21.15.97:6816/1465924719,v1:172.21.15.97:6817/1465924719]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:47:04.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:04 smithi049 ceph-mon[96530]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:47:04.767 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:47:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:47:04.508+0000 7f260ad7a700 -1 osd.8 0 waiting for initial osdmap 2023-07-20T17:47:04.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:47:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:47:04.520+0000 7f2605b6e700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:47:04.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:04 smithi079 ceph-mon[98036]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:04.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:04 smithi079 ceph-mon[98036]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T17:47:04.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:04 smithi079 ceph-mon[98036]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T17:47:04.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:04 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:04.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:04 smithi079 ceph-mon[98036]: from='osd.8 [v2:172.21.15.97:6816/1465924719,v1:172.21.15.97:6817/1465924719]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:47:04.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:04 smithi079 ceph-mon[98036]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:47:04.927 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T17:47:04.928 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T17:47:04.928 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2023-07-20T17:47:04.928 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2023-07-20T17:47:04.929 INFO:teuthology.orchestra.run.smithi049.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00844475 s, 1.2 GB/s 2023-07-20T17:47:04.929 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2023-07-20T17:47:05.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi049:vg_nvme/lv_1 2023-07-20T17:47:05.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:47:05.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T17:47:05.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:05.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:05.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:05 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:47:05.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T17:47:05.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:05.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:05.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:05.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:05.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:05.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:05.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:05.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:05 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:47:05.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T17:47:05.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:05.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:05.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:05.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:05.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:05.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:05.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:05.820 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:05 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:05.872 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:47:06.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:06 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:47:06.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:06 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:47:06.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:06 smithi049 ceph-mon[96530]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:06.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:06 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi097 2023-07-20T17:47:06.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:06 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi097 to 5722M 2023-07-20T17:47:06.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:06.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:06 smithi049 ceph-mon[96530]: osd.8 [v2:172.21.15.97:6816/1465924719,v1:172.21.15.97:6817/1465924719] boot 2023-07-20T17:47:06.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:06 smithi049 ceph-mon[96530]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T17:47:06.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:06.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:06 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:47:06.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:06 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:47:06.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:06 smithi097 ceph-mon[98398]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:06.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:06 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi097 2023-07-20T17:47:06.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:06 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi097 to 5722M 2023-07-20T17:47:06.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:06.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:06 smithi097 ceph-mon[98398]: osd.8 [v2:172.21.15.97:6816/1465924719,v1:172.21.15.97:6817/1465924719] boot 2023-07-20T17:47:06.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:06 smithi097 ceph-mon[98398]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T17:47:06.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:06.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:06 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:47:06.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:06 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:47:06.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:06 smithi079 ceph-mon[98036]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T17:47:06.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:06 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi097 2023-07-20T17:47:06.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:06 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi097 to 5722M 2023-07-20T17:47:06.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:06.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:06 smithi079 ceph-mon[98036]: osd.8 [v2:172.21.15.97:6816/1465924719,v1:172.21.15.97:6817/1465924719] boot 2023-07-20T17:47:06.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:06 smithi079 ceph-mon[98036]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T17:47:06.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T17:47:07.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.026+0000 7efdbaebc700 1 -- 172.21.15.49:0/3998370352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb40ff9d0 msgr2=0x7efdb40ffe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:07.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.026+0000 7efdbaebc700 1 --2- 172.21.15.49:0/3998370352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb40ff9d0 0x7efdb40ffe30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efda0009990 tx=0x7efda0009ca0 comp rx=0 tx=0).stop 2023-07-20T17:47:07.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.026+0000 7efdbaebc700 1 -- 172.21.15.49:0/3998370352 shutdown_connections 2023-07-20T17:47:07.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.026+0000 7efdbaebc700 1 --2- 172.21.15.49:0/3998370352 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdb4100c90 0x7efdb41010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:07.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.026+0000 7efdbaebc700 1 --2- 172.21.15.49:0/3998370352 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efdb4100370 0x7efdb4107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:07.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.026+0000 7efdbaebc700 1 --2- 172.21.15.49:0/3998370352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb40ff9d0 0x7efdb40ffe30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:07.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.026+0000 7efdbaebc700 1 -- 172.21.15.49:0/3998370352 >> 172.21.15.49:0/3998370352 conn(0x7efdb40fa420 msgr2=0x7efdb40fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:47:07.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.026+0000 7efdbaebc700 1 -- 172.21.15.49:0/3998370352 shutdown_connections 2023-07-20T17:47:07.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.026+0000 7efdbaebc700 1 -- 172.21.15.49:0/3998370352 wait complete. 2023-07-20T17:47:07.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.027+0000 7efdbaebc700 1 Processor -- start 2023-07-20T17:47:07.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.027+0000 7efdbaebc700 1 -- start start 2023-07-20T17:47:07.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.027+0000 7efdbaebc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb40ff9d0 0x7efdb40ff440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.027+0000 7efdbaebc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efdb4100370 0x7efdb419b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.027+0000 7efdbaebc700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdb4100c90 0x7efdb419d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.027+0000 7efdbaebc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb419dfb0 con 0x7efdb40ff9d0 2023-07-20T17:47:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.027+0000 7efdbaebc700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb419e120 con 0x7efdb4100370 2023-07-20T17:47:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.027+0000 7efdbaebc700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb419e2e0 con 0x7efdb4100c90 2023-07-20T17:47:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.027+0000 7efdb8c58700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb40ff9d0 0x7efdb40ff440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.027+0000 7efdb8c58700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb40ff9d0 0x7efdb40ff440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35924/0 (socket says 172.21.15.49:35924) 2023-07-20T17:47:07.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.027+0000 7efdb8c58700 1 -- 172.21.15.49:0/1645766776 learned_addr learned my addr 172.21.15.49:0/1645766776 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:47:07.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb3fff700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efdb4100370 0x7efdb419b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:07.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb9459700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdb4100c90 0x7efdb419d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:07.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb8c58700 1 -- 172.21.15.49:0/1645766776 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdb4100c90 msgr2=0x7efdb419d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:07.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb8c58700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdb4100c90 0x7efdb419d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:07.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb8c58700 1 -- 172.21.15.49:0/1645766776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efdb4100370 msgr2=0x7efdb419b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:07.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb8c58700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efdb4100370 0x7efdb419b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:07.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb8c58700 1 -- 172.21.15.49:0/1645766776 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda00095f0 con 0x7efdb40ff9d0 2023-07-20T17:47:07.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb3fff700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efdb4100370 0x7efdb419b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T17:47:07.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb8c58700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb40ff9d0 0x7efdb40ff440 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efda800d980 tx=0x7efda800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:47:07.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb1ffb700 1 -- 172.21.15.49:0/1645766776 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda800de60 con 0x7efdb40ff9d0 2023-07-20T17:47:07.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb1ffb700 1 -- 172.21.15.49:0/1645766776 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efda80044e0 con 0x7efdb40ff9d0 2023-07-20T17:47:07.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdbaebc700 1 -- 172.21.15.49:0/1645766776 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb419e5c0 con 0x7efdb40ff9d0 2023-07-20T17:47:07.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdb1ffb700 1 -- 172.21.15.49:0/1645766776 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8003be0 con 0x7efdb40ff9d0 2023-07-20T17:47:07.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.028+0000 7efdbaebc700 1 -- 172.21.15.49:0/1645766776 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb419eb10 con 0x7efdb40ff9d0 2023-07-20T17:47:07.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.029+0000 7efdbaebc700 1 -- 172.21.15.49:0/1645766776 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb4106e00 con 0x7efdb40ff9d0 2023-07-20T17:47:07.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.030+0000 7efdb1ffb700 1 -- 172.21.15.49:0/1645766776 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efda8009bb0 con 0x7efdb40ff9d0 2023-07-20T17:47:07.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.031+0000 7efdb1ffb700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7efda40a11d0 0x7efda40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:07.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.031+0000 7efdb3fff700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7efda40a11d0 0x7efda40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:07.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.031+0000 7efdb1ffb700 1 -- 172.21.15.49:0/1645766776 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5858+0+0 (secure 0 0 0) 0x7efda80cbcb0 con 0x7efdb40ff9d0 2023-07-20T17:47:07.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.031+0000 7efdb3fff700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7efda40a11d0 0x7efda40a3690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efda000b5c0 tx=0x7efda0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:47:07.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.036+0000 7efdb1ffb700 1 -- 172.21.15.49:0/1645766776 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efda8095d90 con 0x7efdb40ff9d0 2023-07-20T17:47:07.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:07.242+0000 7efdbaebc700 1 -- 172.21.15.49:0/1645766776 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7efdb4060bb0 con 0x7efda40a11d0 2023-07-20T17:47:07.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:07 smithi079 ceph-mon[98036]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T17:47:07.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:07 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:47:07.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:07 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:47:07.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:07 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:07.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:07 smithi049 ceph-mon[96530]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T17:47:07.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:07 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:47:07.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:07 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:47:07.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:07 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:08.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:07 smithi097 ceph-mon[98398]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T17:47:08.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:07 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:47:08.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:07 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:47:08.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:07 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:08.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:08 smithi049 ceph-mon[96530]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:08.650 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:08 smithi049 ceph-mon[96530]: from='client.14409 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:47:08.650 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:08 smithi049 ceph-mon[96530]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T17:47:08.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:08 smithi079 ceph-mon[98036]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:08.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:08 smithi079 ceph-mon[98036]: from='client.14409 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:47:08.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:08 smithi079 ceph-mon[98036]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T17:47:09.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:08 smithi097 ceph-mon[98398]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:09.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:08 smithi097 ceph-mon[98398]: from='client.14409 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:47:09.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:08 smithi097 ceph-mon[98398]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T17:47:09.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:09 smithi079 ceph-mon[98036]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T17:47:09.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:09 smithi049 ceph-mon[96530]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T17:47:10.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:09 smithi097 ceph-mon[98398]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T17:47:10.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:10 smithi079 ceph-mon[98036]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:10.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:10 smithi079 ceph-mon[98036]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T17:47:10.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:10 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/238368859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0c07e94-8039-4e41-85b7-23344124687b"}]: dispatch 2023-07-20T17:47:10.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:10 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/238368859' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0c07e94-8039-4e41-85b7-23344124687b"}]': finished 2023-07-20T17:47:10.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:10 smithi079 ceph-mon[98036]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T17:47:10.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:10 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:10.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:10 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/2457133474' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:47:10.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:10 smithi049 ceph-mon[96530]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:10.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:10 smithi049 ceph-mon[96530]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T17:47:10.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:10 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/238368859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0c07e94-8039-4e41-85b7-23344124687b"}]: dispatch 2023-07-20T17:47:10.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:10 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/238368859' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0c07e94-8039-4e41-85b7-23344124687b"}]': finished 2023-07-20T17:47:10.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:10 smithi049 ceph-mon[96530]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T17:47:10.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:10 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:10.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:10 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2457133474' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:47:11.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:10 smithi097 ceph-mon[98398]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:11.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:10 smithi097 ceph-mon[98398]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T17:47:11.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:10 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/238368859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0c07e94-8039-4e41-85b7-23344124687b"}]: dispatch 2023-07-20T17:47:11.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:10 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/238368859' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0c07e94-8039-4e41-85b7-23344124687b"}]': finished 2023-07-20T17:47:11.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:10 smithi097 ceph-mon[98398]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T17:47:11.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:10 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:11.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:10 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/2457133474' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:47:12.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:12 smithi079 ceph-mon[98036]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:12.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:12 smithi049 ceph-mon[96530]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:13.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:12 smithi097 ceph-mon[98398]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:14.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:14 smithi079 ceph-mon[98036]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2023-07-20T17:47:14.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:14 smithi049 ceph-mon[96530]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2023-07-20T17:47:15.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:14 smithi097 ceph-mon[98398]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2023-07-20T17:47:16.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:16 smithi079 ceph-mon[98036]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2023-07-20T17:47:16.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:16 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T17:47:16.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:16 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:16.863 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:16 smithi049 ceph-mon[96530]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2023-07-20T17:47:16.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:16 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T17:47:16.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:16 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:17.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:16 smithi097 ceph-mon[98398]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2023-07-20T17:47:17.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:16 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T17:47:17.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:16 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:17.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:17 smithi079 ceph-mon[98036]: Deploying daemon osd.9 on smithi049 2023-07-20T17:47:17.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:17 smithi049 ceph-mon[96530]: Deploying daemon osd.9 on smithi049 2023-07-20T17:47:18.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:17 smithi097 ceph-mon[98398]: Deploying daemon osd.9 on smithi049 2023-07-20T17:47:18.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:18 smithi049 ceph-mon[96530]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T17:47:18.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:18 smithi079 ceph-mon[98036]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T17:47:19.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:18 smithi097 ceph-mon[98398]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T17:47:20.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:20 smithi079 ceph-mon[98036]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-07-20T17:47:20.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:20 smithi049 ceph-mon[96530]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-07-20T17:47:21.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:20 smithi097 ceph-mon[98398]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-07-20T17:47:21.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:21 smithi049 ceph-mon[96530]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-07-20T17:47:21.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:21.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:21.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:22.267 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:21 smithi097 ceph-mon[98398]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-07-20T17:47:22.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:22.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:22.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:22.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:21 smithi079 ceph-mon[98036]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-07-20T17:47:22.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:22.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:22.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:23.599 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 9 on host 'smithi049' 2023-07-20T17:47:23.600 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.597+0000 7efdb1ffb700 1 -- 172.21.15.49:0/1645766776 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efdb4060bb0 con 0x7efda40a11d0 2023-07-20T17:47:23.601 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 -- 172.21.15.49:0/1645766776 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7efda40a11d0 msgr2=0x7efda40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:23.602 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7efda40a11d0 0x7efda40a3690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efda000b5c0 tx=0x7efda0005fb0 comp rx=0 tx=0).stop 2023-07-20T17:47:23.602 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 -- 172.21.15.49:0/1645766776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb40ff9d0 msgr2=0x7efdb40ff440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:23.602 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb40ff9d0 0x7efdb40ff440 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efda800d980 tx=0x7efda800dd40 comp rx=0 tx=0).stop 2023-07-20T17:47:23.602 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 -- 172.21.15.49:0/1645766776 shutdown_connections 2023-07-20T17:47:23.602 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7efda40a11d0 0x7efda40a3690 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:23.602 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efdb4100c90 0x7efdb419d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:23.602 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb40ff9d0 0x7efdb40ff440 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:23.602 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 --2- 172.21.15.49:0/1645766776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efdb4100370 0x7efdb419b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:23.603 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 -- 172.21.15.49:0/1645766776 >> 172.21.15.49:0/1645766776 conn(0x7efdb40fa420 msgr2=0x7efdb4103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:47:23.603 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 -- 172.21.15.49:0/1645766776 shutdown_connections 2023-07-20T17:47:23.603 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:47:23.599+0000 7efdbaebc700 1 -- 172.21.15.49:0/1645766776 wait complete. 2023-07-20T17:47:24.077 DEBUG:teuthology.orchestra.run.smithi049:osd.9> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.9.service 2023-07-20T17:47:24.079 INFO:tasks.cephadm:Deploying osd.10 on smithi079 with /dev/vg_nvme/lv_1... 2023-07-20T17:47:24.080 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T17:47:24.113 INFO:journalctl@ceph.osd.9.smithi049.stdout:-- Logs begin at Thu 2023-07-20 17:16:49 UTC. -- 2023-07-20T17:47:24.642 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T17:47:24.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:24 smithi097 ceph-mon[98398]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T17:47:24.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:24 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:24 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:24 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:24.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:24 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:24.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:24 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:24 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:24 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:24 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:24.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:24 smithi049 ceph-mon[96530]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T17:47:24.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:24 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:24 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:24 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:24.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:24 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:24.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:24 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:24 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:24 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:24 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:24.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:24 smithi079 ceph-mon[98036]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T17:47:24.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:24 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:24 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:24 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:24.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:24 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:24.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:24 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:24 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:24 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:24.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:24 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:25.046 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:47:24 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:47:24.798+0000 7eff2c435640 -1 osd.9 0 log_to_monitors true 2023-07-20T17:47:25.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:25 smithi049 ceph-mon[96530]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T17:47:25.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:25 smithi097 ceph-mon[98398]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T17:47:25.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:25 smithi079 ceph-mon[98036]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T17:47:26.000 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T17:47:26.000 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T17:47:26.000 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2023-07-20T17:47:26.001 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2023-07-20T17:47:26.001 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110709 s, 947 MB/s 2023-07-20T17:47:26.001 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2023-07-20T17:47:26.382 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi079:vg_nvme/lv_1 2023-07-20T17:47:26.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:26.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T17:47:26.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T17:47:26.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:26.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:47:26.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:26.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:26.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:26.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:26.571 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:26.571 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:26 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:26.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:26.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T17:47:26.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T17:47:26.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:26.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:47:26.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:26.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:26.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:26.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:26.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:26.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:26 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:26.935 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T17:47:26.990 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:47:26.508+0000 7eff21aad700 -1 osd.9 0 waiting for initial osdmap 2023-07-20T17:47:26.990 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:47:26.519+0000 7eff1d0a2700 -1 osd.9 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:47:26.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:26.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T17:47:26.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T17:47:26.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:26.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2023-07-20T17:47:26.996 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:26.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:26.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:26.999 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:26.999 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:26.999 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:27.000 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:26 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:27.803 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:27 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi049 2023-07-20T17:47:27.803 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:27 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi049 to 4292M 2023-07-20T17:47:27.804 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:27 smithi079 ceph-mon[98036]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:47:27.804 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:27 smithi079 ceph-mon[98036]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T17:47:27.804 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:27 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:27.804 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:27 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:27.804 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:27 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:27.968 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.965+0000 7f58e7a67700 1 -- 172.21.15.79:0/1949149910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58e01009a0 msgr2=0x7f58e01072c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:27.968 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.965+0000 7f58e7a67700 1 --2- 172.21.15.79:0/1949149910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58e01009a0 0x7f58e01072c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f58d800b0a0 tx=0x7f58d800b3b0 comp rx=0 tx=0).stop 2023-07-20T17:47:27.969 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 -- 172.21.15.79:0/1949149910 shutdown_connections 2023-07-20T17:47:27.969 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 --2- 172.21.15.79:0/1949149910 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e00fee80 0x7f58e00ff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:27.969 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 --2- 172.21.15.79:0/1949149910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58e01009a0 0x7f58e01072c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:27.969 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 --2- 172.21.15.79:0/1949149910 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58e00fffe0 0x7f58e0100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:27.969 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 -- 172.21.15.79:0/1949149910 >> 172.21.15.79:0/1949149910 conn(0x7f58e00fa420 msgr2=0x7f58e00fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:47:27.969 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 -- 172.21.15.79:0/1949149910 shutdown_connections 2023-07-20T17:47:27.970 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 -- 172.21.15.79:0/1949149910 wait complete. 2023-07-20T17:47:27.971 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 Processor -- start 2023-07-20T17:47:27.971 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 -- start start 2023-07-20T17:47:27.971 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e00fee80 0x7f58e0103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:27.972 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58e00fffe0 0x7f58e0101660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:27.972 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.966+0000 7f58e7a67700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58e01009a0 0x7f58e0101ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:27.972 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e7a67700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e0102320 con 0x7f58e01009a0 2023-07-20T17:47:27.972 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e7a67700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e0102490 con 0x7f58e00fffe0 2023-07-20T17:47:27.972 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e7a67700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e01a0640 con 0x7f58e00fee80 2023-07-20T17:47:27.972 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e5002700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58e00fffe0 0x7f58e0101660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:27.972 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e5002700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58e00fffe0 0x7f58e0101660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48068/0 (socket says 172.21.15.79:48068) 2023-07-20T17:47:27.973 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e5002700 1 -- 172.21.15.79:0/420728119 learned_addr learned my addr 172.21.15.79:0/420728119 (peer_addr_for_me v2:172.21.15.79:0/0) 2023-07-20T17:47:27.973 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e5803700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e00fee80 0x7f58e0103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:27.973 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e6004700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58e01009a0 0x7f58e0101ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:27.973 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e5002700 1 -- 172.21.15.79:0/420728119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e00fee80 msgr2=0x7f58e0103010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:27.973 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e5002700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e00fee80 0x7f58e0103010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:27.973 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e5002700 1 -- 172.21.15.79:0/420728119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58e01009a0 msgr2=0x7f58e0101ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:27.973 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e5002700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58e01009a0 0x7f58e0101ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:27.974 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e5002700 1 -- 172.21.15.79:0/420728119 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d8009d00 con 0x7f58e00fffe0 2023-07-20T17:47:27.974 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e5002700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58e00fffe0 0x7f58e0101660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f58dc00eab0 tx=0x7f58dc00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:47:27.974 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58d6ffd700 1 -- 172.21.15.79:0/420728119 <== mon.2 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc00ccd0 con 0x7f58e00fffe0 2023-07-20T17:47:27.974 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e7a67700 1 -- 172.21.15.79:0/420728119 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e01a0920 con 0x7f58e00fffe0 2023-07-20T17:47:27.975 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58d6ffd700 1 -- 172.21.15.79:0/420728119 <== mon.2 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58dc00ce30 con 0x7f58e00fffe0 2023-07-20T17:47:27.975 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58d6ffd700 1 -- 172.21.15.79:0/420728119 <== mon.2 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc0157c0 con 0x7f58e00fffe0 2023-07-20T17:47:27.975 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.967+0000 7f58e7a67700 1 -- 172.21.15.79:0/420728119 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e01a0e20 con 0x7f58e00fffe0 2023-07-20T17:47:27.975 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.968+0000 7f58e7a67700 1 -- 172.21.15.79:0/420728119 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e0109ee0 con 0x7f58e00fffe0 2023-07-20T17:47:27.976 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.969+0000 7f58d6ffd700 1 -- 172.21.15.79:0/420728119 <== mon.2 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f58dc0163e0 con 0x7f58e00fffe0 2023-07-20T17:47:27.976 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.970+0000 7f58d6ffd700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58cc0a1150 0x7f58cc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:27.977 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.970+0000 7f58d6ffd700 1 -- 172.21.15.79:0/420728119 <== mon.2 v2:172.21.15.79:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f58dc0cc590 con 0x7f58e00fffe0 2023-07-20T17:47:27.977 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.970+0000 7f58e5803700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58cc0a1150 0x7f58cc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:27.977 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.971+0000 7f58e5803700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58cc0a1150 0x7f58cc0a3610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009c60 tx=0x7f58d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:47:27.977 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:27.973+0000 7f58d6ffd700 1 -- 172.21.15.79:0/420728119 <== mon.2 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f58dc0964e0 con 0x7f58e00fffe0 2023-07-20T17:47:27.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:27 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi049 2023-07-20T17:47:27.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:27 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi049 to 4292M 2023-07-20T17:47:27.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:27 smithi049 ceph-mon[96530]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:47:27.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:27 smithi049 ceph-mon[96530]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T17:47:27.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:27 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:27.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:27 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:27.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:27 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:28.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:27 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi049 2023-07-20T17:47:28.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:27 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi049 to 4292M 2023-07-20T17:47:28.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:27 smithi097 ceph-mon[98398]: from='osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2023-07-20T17:47:28.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:27 smithi097 ceph-mon[98398]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T17:47:28.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:27 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:28.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:27 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:28.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:27 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:28.173 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:28.170+0000 7f58e7a67700 1 -- 172.21.15.79:0/420728119 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f58e0106de0 con 0x7f58cc0a1150 2023-07-20T17:47:28.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:28 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:47:28.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:28 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:47:28.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:28 smithi079 ceph-mon[98036]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:28.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:28 smithi079 ceph-mon[98036]: osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131] boot 2023-07-20T17:47:28.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:28 smithi079 ceph-mon[98036]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T17:47:28.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:28.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:47:28.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:47:28.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:28.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:28 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:47:28.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:28 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:47:28.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:28 smithi049 ceph-mon[96530]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:28.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:28 smithi049 ceph-mon[96530]: osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131] boot 2023-07-20T17:47:28.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:28 smithi049 ceph-mon[96530]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T17:47:28.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:28.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:47:28.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:47:28.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:29.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:28 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:47:29.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:28 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:47:29.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:28 smithi097 ceph-mon[98398]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T17:47:29.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:28 smithi097 ceph-mon[98398]: osd.9 [v2:172.21.15.49:6826/845176131,v1:172.21.15.49:6827/845176131] boot 2023-07-20T17:47:29.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:28 smithi097 ceph-mon[98398]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T17:47:29.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T17:47:29.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:47:29.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:47:29.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:29.790 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:29 smithi079 ceph-mon[98036]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:47:29.790 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:29 smithi079 ceph-mon[98036]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T17:47:29.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:29 smithi049 ceph-mon[96530]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:47:29.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:29 smithi049 ceph-mon[96530]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T17:47:30.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:29 smithi097 ceph-mon[98398]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:47:30.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:29 smithi097 ceph-mon[98398]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T17:47:30.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:30 smithi079 ceph-mon[98036]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:30.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:30 smithi079 ceph-mon[98036]: from='client.? 172.21.15.79:0/2087212420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864"}]: dispatch 2023-07-20T17:47:30.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:30 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864"}]: dispatch 2023-07-20T17:47:30.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:30 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864"}]': finished 2023-07-20T17:47:30.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:30 smithi079 ceph-mon[98036]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T17:47:30.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:30 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:30.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:30 smithi049 ceph-mon[96530]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:30.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:30 smithi049 ceph-mon[96530]: from='client.? 172.21.15.79:0/2087212420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864"}]: dispatch 2023-07-20T17:47:30.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:30 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864"}]: dispatch 2023-07-20T17:47:30.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:30 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864"}]': finished 2023-07-20T17:47:30.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:30 smithi049 ceph-mon[96530]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T17:47:30.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:30 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:31.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:30 smithi097 ceph-mon[98398]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:31.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:30 smithi097 ceph-mon[98398]: from='client.? 172.21.15.79:0/2087212420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864"}]: dispatch 2023-07-20T17:47:31.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:30 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864"}]: dispatch 2023-07-20T17:47:31.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:30 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864"}]': finished 2023-07-20T17:47:31.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:30 smithi097 ceph-mon[98398]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T17:47:31.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:31.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:31 smithi079 ceph-mon[98036]: from='client.? 172.21.15.79:0/3175788602' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:47:31.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:31 smithi049 ceph-mon[96530]: from='client.? 172.21.15.79:0/3175788602' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:47:32.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:31 smithi097 ceph-mon[98398]: from='client.? 172.21.15.79:0/3175788602' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:47:32.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:32 smithi079 ceph-mon[98036]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:32.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:32 smithi049 ceph-mon[96530]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:33.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:32 smithi097 ceph-mon[98398]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:34.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:34 smithi079 ceph-mon[98036]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:34.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:34 smithi049 ceph-mon[96530]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:35.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:34 smithi097 ceph-mon[98398]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:36.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:36 smithi079 ceph-mon[98036]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:36.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:36 smithi049 ceph-mon[96530]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:37.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:36 smithi097 ceph-mon[98398]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:37.744 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:37 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T17:47:37.745 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:37 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:37.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:37 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T17:47:37.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:37 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:38.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:37 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T17:47:38.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:37 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:38.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:38 smithi079 ceph-mon[98036]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:38.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:38 smithi079 ceph-mon[98036]: Deploying daemon osd.10 on smithi079 2023-07-20T17:47:38.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:38 smithi049 ceph-mon[96530]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:38.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:38 smithi049 ceph-mon[96530]: Deploying daemon osd.10 on smithi079 2023-07-20T17:47:39.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:38 smithi097 ceph-mon[98398]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:39.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:38 smithi097 ceph-mon[98398]: Deploying daemon osd.10 on smithi079 2023-07-20T17:47:40.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:40 smithi079 ceph-mon[98036]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:40.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:40 smithi049 ceph-mon[96530]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:41.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:40 smithi097 ceph-mon[98398]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:42.302 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:42 smithi079 ceph-mon[98036]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:42.303 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:42.303 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:42.303 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:42.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:42 smithi049 ceph-mon[96530]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:42.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:42.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:42.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:42.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:42 smithi097 ceph-mon[98398]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:42.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:42.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:42.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:43.740 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 10 on host 'smithi079' 2023-07-20T17:47:43.740 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.738+0000 7f58d6ffd700 1 -- 172.21.15.79:0/420728119 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f58e0106de0 con 0x7f58cc0a1150 2023-07-20T17:47:43.742 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 -- 172.21.15.79:0/420728119 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58cc0a1150 msgr2=0x7f58cc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:43.742 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58cc0a1150 0x7f58cc0a3610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009c60 tx=0x7f58d0009260 comp rx=0 tx=0).stop 2023-07-20T17:47:43.742 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 -- 172.21.15.79:0/420728119 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58e00fffe0 msgr2=0x7f58e0101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:43.742 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58e00fffe0 0x7f58e0101660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f58dc00eab0 tx=0x7f58dc00edc0 comp rx=0 tx=0).stop 2023-07-20T17:47:43.743 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 -- 172.21.15.79:0/420728119 shutdown_connections 2023-07-20T17:47:43.743 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58cc0a1150 0x7f58cc0a3610 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:43.743 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58e00fee80 0x7f58e0103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:43.743 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58e01009a0 0x7f58e0101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:43.743 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 --2- 172.21.15.79:0/420728119 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58e00fffe0 0x7f58e0101660 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:43.743 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 -- 172.21.15.79:0/420728119 >> 172.21.15.79:0/420728119 conn(0x7f58e00fa420 msgr2=0x7f58e01056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:47:43.743 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 -- 172.21.15.79:0/420728119 shutdown_connections 2023-07-20T17:47:43.744 INFO:teuthology.orchestra.run.smithi079.stderr:2023-07-20T17:47:43.741+0000 7f58e7a67700 1 -- 172.21.15.79:0/420728119 wait complete. 2023-07-20T17:47:44.114 DEBUG:teuthology.orchestra.run.smithi079:osd.10> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.10.service 2023-07-20T17:47:44.117 INFO:tasks.cephadm:Deploying osd.11 on smithi097 with /dev/vg_nvme/lv_1... 2023-07-20T17:47:44.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T17:47:44.146 INFO:journalctl@ceph.osd.10.smithi079.stdout:-- Logs begin at Thu 2023-07-20 17:15:42 UTC. -- 2023-07-20T17:47:44.685 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:47:44.931 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:44 smithi097 ceph-mon[98398]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:44.932 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:44.932 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:44.932 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:44.932 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:44.935 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:44 smithi079 ceph-mon[98036]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:44.936 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:44.936 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:44.936 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:44.936 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:44.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:44 smithi049 ceph-mon[96530]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:44.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:44.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:44.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:44.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:45.317 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:47:45 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:47:45.086+0000 7f0508728640 -1 osd.10 0 log_to_monitors true 2023-07-20T17:47:45.739 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:45 smithi097 ceph-mon[98398]: from='osd.10 [v2:172.21.15.79:6824/935518328,v1:172.21.15.79:6825/935518328]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T17:47:45.740 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:45 smithi097 ceph-mon[98398]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T17:47:45.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:45 smithi079 ceph-mon[98036]: from='osd.10 [v2:172.21.15.79:6824/935518328,v1:172.21.15.79:6825/935518328]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T17:47:45.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:45 smithi079 ceph-mon[98036]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T17:47:45.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:45 smithi049 ceph-mon[96530]: from='osd.10 [v2:172.21.15.79:6824/935518328,v1:172.21.15.79:6825/935518328]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T17:47:45.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:45 smithi049 ceph-mon[96530]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T17:47:46.210 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T17:47:46.210 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T17:47:46.210 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2023-07-20T17:47:46.211 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2023-07-20T17:47:46.211 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012946 s, 810 MB/s 2023-07-20T17:47:46.211 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2023-07-20T17:47:46.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi097:vg_nvme/lv_1 2023-07-20T17:47:46.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:46.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T17:47:46.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T17:47:46.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:46.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='osd.10 [v2:172.21.15.79:6824/935518328,v1:172.21.15.79:6825/935518328]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:47:46.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:47:46.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:46.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:46.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:46.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:46.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:46.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:46.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:46.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:46.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:46.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:46.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:46.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:46.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:46.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:47.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:47.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T17:47:47.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T17:47:47.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:47.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='osd.10 [v2:172.21.15.79:6824/935518328,v1:172.21.15.79:6825/935518328]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:47:47.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:47:47.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:47.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:47.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:47.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:47.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:47.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:47.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:47.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:47.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:47.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:47.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:47.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:47.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:47.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:47.067 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:47:46.689+0000 7f04fdda0700 -1 osd.10 0 waiting for initial osdmap 2023-07-20T17:47:47.067 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:47:46.700+0000 7f04f638f700 -1 osd.10 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:47:47.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:47.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T17:47:47.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T17:47:47.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:47.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='osd.10 [v2:172.21.15.79:6824/935518328,v1:172.21.15.79:6825/935518328]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:47:47.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2023-07-20T17:47:47.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:47.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:47.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:47.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:47.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:47.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:47:47.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:47.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:47.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:47.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:47.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:47:47.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:47.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:47:47.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:47:47.183 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:47:47.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:47 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi079 2023-07-20T17:47:47.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:47 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi079 to 4292M 2023-07-20T17:47:47.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:47 smithi049 ceph-mon[96530]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:47:47.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:47 smithi049 ceph-mon[96530]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T17:47:47.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:47 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:47.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:47 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:47.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:47 smithi049 ceph-mon[96530]: osd.10 [v2:172.21.15.79:6824/935518328,v1:172.21.15.79:6825/935518328] boot 2023-07-20T17:47:47.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:47 smithi049 ceph-mon[96530]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T17:47:47.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:47 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:48.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:47 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi079 2023-07-20T17:47:48.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:47 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi079 to 4292M 2023-07-20T17:47:48.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:47 smithi097 ceph-mon[98398]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:47:48.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:47 smithi097 ceph-mon[98398]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T17:47:48.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:47 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:48.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:47 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:48.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:47 smithi097 ceph-mon[98398]: osd.10 [v2:172.21.15.79:6824/935518328,v1:172.21.15.79:6825/935518328] boot 2023-07-20T17:47:48.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:47 smithi097 ceph-mon[98398]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T17:47:48.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:47 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:48.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:47 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi079 2023-07-20T17:47:48.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:47 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi079 to 4292M 2023-07-20T17:47:48.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:47 smithi079 ceph-mon[98036]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2023-07-20T17:47:48.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:47 smithi079 ceph-mon[98036]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T17:47:48.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:47 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:48.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:47 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:48.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:47 smithi079 ceph-mon[98036]: osd.10 [v2:172.21.15.79:6824/935518328,v1:172.21.15.79:6825/935518328] boot 2023-07-20T17:47:48.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:47 smithi079 ceph-mon[98036]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T17:47:48.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:47 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T17:47:48.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 -- 172.21.15.97:0/2417362813 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff10c101de0 msgr2=0x7ff10c102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:48.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 --2- 172.21.15.97:0/2417362813 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff10c101de0 0x7ff10c102200 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff108009990 tx=0x7ff108009ca0 comp rx=0 tx=0).stop 2023-07-20T17:47:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 -- 172.21.15.97:0/2417362813 shutdown_connections 2023-07-20T17:47:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 --2- 172.21.15.97:0/2417362813 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff10c101de0 0x7ff10c102200 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 --2- 172.21.15.97:0/2417362813 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff10c0ff200 0x7ff10c108dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 --2- 172.21.15.97:0/2417362813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff10c0fe860 0x7ff10c0fecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 -- 172.21.15.97:0/2417362813 >> 172.21.15.97:0/2417362813 conn(0x7ff10c0fa420 msgr2=0x7ff10c0fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:47:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 -- 172.21.15.97:0/2417362813 shutdown_connections 2023-07-20T17:47:48.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 -- 172.21.15.97:0/2417362813 wait complete. 2023-07-20T17:47:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 Processor -- start 2023-07-20T17:47:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 -- start start 2023-07-20T17:47:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff10c0fe860 0x7ff10c1014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff10c0ff200 0x7ff10c0ffb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff10c101de0 0x7ff10c100070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c1007f0 con 0x7ff10c0ff200 2023-07-20T17:47:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c100960 con 0x7ff10c0fe860 2023-07-20T17:47:48.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.519+0000 7ff11421a700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c1a0430 con 0x7ff10c101de0 2023-07-20T17:47:48.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff1127b7700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff10c101de0 0x7ff10c100070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:48.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff1127b7700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff10c101de0 0x7ff10c100070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32772/0 (socket says 172.21.15.97:32772) 2023-07-20T17:47:48.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff1127b7700 1 -- 172.21.15.97:0/1814789095 learned_addr learned my addr 172.21.15.97:0/1814789095 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-07-20T17:47:48.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff111fb6700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff10c0fe860 0x7ff10c1014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:48.525 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff1117b5700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff10c0ff200 0x7ff10c0ffb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:48.526 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff1127b7700 1 -- 172.21.15.97:0/1814789095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff10c0fe860 msgr2=0x7ff10c1014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:48.526 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff1127b7700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff10c0fe860 0x7ff10c1014e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:48.526 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff1127b7700 1 -- 172.21.15.97:0/1814789095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff10c0ff200 msgr2=0x7ff10c0ffb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:47:48.527 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff1127b7700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff10c0ff200 0x7ff10c0ffb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:47:48.527 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff1127b7700 1 -- 172.21.15.97:0/1814789095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1080095f0 con 0x7ff10c101de0 2023-07-20T17:47:48.527 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff1127b7700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff10c101de0 0x7ff10c100070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff104007d70 tx=0x7ff10400a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:47:48.527 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff102ffd700 1 -- 172.21.15.97:0/1814789095 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10400ad40 con 0x7ff10c101de0 2023-07-20T17:47:48.527 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff102ffd700 1 -- 172.21.15.97:0/1814789095 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1040128e0 con 0x7ff10c101de0 2023-07-20T17:47:48.527 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff11421a700 1 -- 172.21.15.97:0/1814789095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff10c1a0710 con 0x7ff10c101de0 2023-07-20T17:47:48.527 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff102ffd700 1 -- 172.21.15.97:0/1814789095 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040116b0 con 0x7ff10c101de0 2023-07-20T17:47:48.527 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.520+0000 7ff11421a700 1 -- 172.21.15.97:0/1814789095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10c1a0bc0 con 0x7ff10c101de0 2023-07-20T17:47:48.528 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.521+0000 7ff11421a700 1 -- 172.21.15.97:0/1814789095 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10c107eb0 con 0x7ff10c101de0 2023-07-20T17:47:48.528 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.522+0000 7ff102ffd700 1 -- 172.21.15.97:0/1814789095 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff104012420 con 0x7ff10c101de0 2023-07-20T17:47:48.528 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.523+0000 7ff102ffd700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff0f80a1150 0x7ff0f80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:47:48.528 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.523+0000 7ff102ffd700 1 -- 172.21.15.97:0/1814789095 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6723+0+0 (secure 0 0 0) 0x7ff1040cc990 con 0x7ff10c101de0 2023-07-20T17:47:48.528 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.523+0000 7ff111fb6700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff0f80a1150 0x7ff0f80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:47:48.528 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.523+0000 7ff111fb6700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff0f80a1150 0x7ff0f80a3610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff10800b5c0 tx=0x7ff108005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:47:48.529 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.526+0000 7ff102ffd700 1 -- 172.21.15.97:0/1814789095 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff104096710 con 0x7ff10c101de0 2023-07-20T17:47:48.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:47:48.720+0000 7ff11421a700 1 -- 172.21.15.97:0/1814789095 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff10c060bb0 con 0x7ff0f80a1150 2023-07-20T17:47:48.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:48 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:47:48.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:48 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:47:48.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:48 smithi049 ceph-mon[96530]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:48.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:48 smithi049 ceph-mon[96530]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T17:47:49.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:48 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:47:49.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:48 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:47:49.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:48 smithi097 ceph-mon[98398]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:49.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:48 smithi097 ceph-mon[98398]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T17:47:49.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:48 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:47:49.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:48 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:47:49.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:48 smithi079 ceph-mon[98036]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T17:47:49.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:48 smithi079 ceph-mon[98036]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T17:47:49.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:49 smithi097 ceph-mon[98398]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:47:49.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:49 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:47:49.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:49 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:47:49.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:49 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:49.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:49 smithi097 ceph-mon[98398]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T17:47:49.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:49 smithi049 ceph-mon[96530]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:47:49.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:49 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:47:49.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:49 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:47:49.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:49 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:49.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:49 smithi049 ceph-mon[96530]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T17:47:50.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:49 smithi079 ceph-mon[98036]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:47:50.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:49 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T17:47:50.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:49 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T17:47:50.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:49 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:50.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:49 smithi079 ceph-mon[98036]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T17:47:50.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:50 smithi049 ceph-mon[96530]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:47:51.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:50 smithi097 ceph-mon[98398]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:47:51.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:50 smithi079 ceph-mon[98036]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:47:51.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:51 smithi049 ceph-mon[96530]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T17:47:51.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:51 smithi049 ceph-mon[96530]: from='client.? 172.21.15.97:0/301388647' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ad9939-51b9-481e-b454-8cfc7451065b"}]: dispatch 2023-07-20T17:47:51.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:51 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ad9939-51b9-481e-b454-8cfc7451065b"}]: dispatch 2023-07-20T17:47:51.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:51 smithi049 ceph-mon[96530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2ad9939-51b9-481e-b454-8cfc7451065b"}]': finished 2023-07-20T17:47:51.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:51 smithi049 ceph-mon[96530]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T17:47:51.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:51 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:47:52.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:51 smithi097 ceph-mon[98398]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T17:47:52.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:51 smithi097 ceph-mon[98398]: from='client.? 172.21.15.97:0/301388647' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ad9939-51b9-481e-b454-8cfc7451065b"}]: dispatch 2023-07-20T17:47:52.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:51 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ad9939-51b9-481e-b454-8cfc7451065b"}]: dispatch 2023-07-20T17:47:52.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:51 smithi097 ceph-mon[98398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2ad9939-51b9-481e-b454-8cfc7451065b"}]': finished 2023-07-20T17:47:52.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:51 smithi097 ceph-mon[98398]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T17:47:52.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:51 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:47:52.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:51 smithi079 ceph-mon[98036]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T17:47:52.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:51 smithi079 ceph-mon[98036]: from='client.? 172.21.15.97:0/301388647' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ad9939-51b9-481e-b454-8cfc7451065b"}]: dispatch 2023-07-20T17:47:52.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:51 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ad9939-51b9-481e-b454-8cfc7451065b"}]: dispatch 2023-07-20T17:47:52.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:51 smithi079 ceph-mon[98036]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2ad9939-51b9-481e-b454-8cfc7451065b"}]': finished 2023-07-20T17:47:52.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:51 smithi079 ceph-mon[98036]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T17:47:52.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:51 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:47:52.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:52 smithi049 ceph-mon[96530]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:47:52.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:52 smithi049 ceph-mon[96530]: from='client.? 172.21.15.97:0/3908957485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:47:53.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:52 smithi097 ceph-mon[98398]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:47:53.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:52 smithi097 ceph-mon[98398]: from='client.? 172.21.15.97:0/3908957485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:47:53.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:52 smithi079 ceph-mon[98036]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:47:53.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:52 smithi079 ceph-mon[98036]: from='client.? 172.21.15.97:0/3908957485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T17:47:54.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:54 smithi049 ceph-mon[96530]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-07-20T17:47:55.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:54 smithi097 ceph-mon[98398]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-07-20T17:47:55.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:54 smithi079 ceph-mon[98036]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-07-20T17:47:56.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:56 smithi049 ceph-mon[96530]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T17:47:57.014 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:56 smithi097 ceph-mon[98398]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T17:47:57.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:56 smithi079 ceph-mon[98036]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T17:47:58.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:58 smithi097 ceph-mon[98398]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-20T17:47:58.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T17:47:58.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:58 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:58.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:58 smithi049 ceph-mon[96530]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-20T17:47:58.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T17:47:58.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:58 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:59.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:58 smithi079 ceph-mon[98036]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-20T17:47:59.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T17:47:59.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:58 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:47:59.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:47:59 smithi049 ceph-mon[96530]: Deploying daemon osd.11 on smithi097 2023-07-20T17:48:00.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:47:59 smithi097 ceph-mon[98398]: Deploying daemon osd.11 on smithi097 2023-07-20T17:48:00.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:47:59 smithi079 ceph-mon[98036]: Deploying daemon osd.11 on smithi097 2023-07-20T17:48:00.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:00 smithi049 ceph-mon[96530]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-20T17:48:01.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:00 smithi097 ceph-mon[98398]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-20T17:48:01.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:00 smithi079 ceph-mon[98036]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-20T17:48:02.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:02 smithi049 ceph-mon[96530]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T17:48:02.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:02.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:02.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:02 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:03.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:02 smithi097 ceph-mon[98398]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T17:48:03.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:03.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:03.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:02 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:03.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:02 smithi079 ceph-mon[98036]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T17:48:03.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:03.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:03.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:02 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:04.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:04 smithi049 ceph-mon[96530]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-20T17:48:05.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:04 smithi097 ceph-mon[98398]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-20T17:48:05.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:04 smithi079 ceph-mon[98036]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-20T17:48:05.235 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 11 on host 'smithi097' 2023-07-20T17:48:05.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.232+0000 7ff102ffd700 1 -- 172.21.15.97:0/1814789095 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7ff10c060bb0 con 0x7ff0f80a1150 2023-07-20T17:48:05.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 -- 172.21.15.97:0/1814789095 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff0f80a1150 msgr2=0x7ff0f80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:05.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff0f80a1150 0x7ff0f80a3610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff10800b5c0 tx=0x7ff108005fd0 comp rx=0 tx=0).stop 2023-07-20T17:48:05.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 -- 172.21.15.97:0/1814789095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff10c101de0 msgr2=0x7ff10c100070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:05.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff10c101de0 0x7ff10c100070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff104007d70 tx=0x7ff10400a340 comp rx=0 tx=0).stop 2023-07-20T17:48:05.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 -- 172.21.15.97:0/1814789095 shutdown_connections 2023-07-20T17:48:05.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff0f80a1150 0x7ff0f80a3610 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:05.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff10c101de0 0x7ff10c100070 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:05.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff10c0fe860 0x7ff10c1014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:05.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 --2- 172.21.15.97:0/1814789095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff10c0ff200 0x7ff10c0ffb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:05.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 -- 172.21.15.97:0/1814789095 >> 172.21.15.97:0/1814789095 conn(0x7ff10c0fa420 msgr2=0x7ff10c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:05.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 -- 172.21.15.97:0/1814789095 shutdown_connections 2023-07-20T17:48:05.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:05.234+0000 7ff11421a700 1 -- 172.21.15.97:0/1814789095 wait complete. 2023-07-20T17:48:05.676 DEBUG:teuthology.orchestra.run.smithi097:osd.11> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.11.service 2023-07-20T17:48:05.679 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-20T17:48:05.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T17:48:05.713 INFO:journalctl@ceph.osd.11.smithi097.stdout:-- Logs begin at Thu 2023-07-20 17:15:39 UTC. -- 2023-07-20T17:48:06.260 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:48:06.407 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:06 smithi097 ceph-mon[98398]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:48:06.407 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.407 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.407 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.407 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:06 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:06 smithi049 ceph-mon[96530]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:48:06.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:06 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:06 smithi079 ceph-mon[98036]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:48:06.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:06 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:06.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:48:06 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:48:06.405+0000 7fc5bfc9f640 -1 osd.11 0 log_to_monitors true 2023-07-20T17:48:07.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:07 smithi097 ceph-mon[98398]: from='osd.11 [v2:172.21.15.97:6824/1087893461,v1:172.21.15.97:6825/1087893461]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T17:48:07.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:07 smithi097 ceph-mon[98398]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T17:48:07.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.305+0000 7f9b7fd69700 1 -- 172.21.15.49:0/817967369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b78100900 msgr2=0x7f9b78100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:07.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.305+0000 7f9b7fd69700 1 --2- 172.21.15.49:0/817967369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b78100900 0x7f9b78100d60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0099c0 tx=0x7f9b6c009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:07.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.305+0000 7f9b7fd69700 1 -- 172.21.15.49:0/817967369 shutdown_connections 2023-07-20T17:48:07.309 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.305+0000 7f9b7fd69700 1 --2- 172.21.15.49:0/817967369 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b780ff700 0x7f9b780ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:07.309 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.305+0000 7f9b7fd69700 1 --2- 172.21.15.49:0/817967369 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b781012a0 0x7f9b78107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:07.309 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.305+0000 7f9b7fd69700 1 --2- 172.21.15.49:0/817967369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b78100900 0x7f9b78100d60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:07.309 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.305+0000 7f9b7fd69700 1 -- 172.21.15.49:0/817967369 >> 172.21.15.49:0/817967369 conn(0x7f9b780fac80 msgr2=0x7f9b780fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:07.309 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.306+0000 7f9b7fd69700 1 -- 172.21.15.49:0/817967369 shutdown_connections 2023-07-20T17:48:07.309 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.306+0000 7f9b7fd69700 1 -- 172.21.15.49:0/817967369 wait complete. 2023-07-20T17:48:07.310 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.306+0000 7f9b7fd69700 1 Processor -- start 2023-07-20T17:48:07.311 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.306+0000 7f9b7fd69700 1 -- start start 2023-07-20T17:48:07.311 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7fd69700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b780ff700 0x7f9b78103690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:07.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7fd69700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b78100900 0x7f9b78101ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:07.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7fd69700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b781012a0 0x7f9b78102220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:07.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7fd69700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b781029a0 con 0x7f9b78100900 2023-07-20T17:48:07.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7fd69700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b78102b10 con 0x7f9b780ff700 2023-07-20T17:48:07.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7fd69700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b781a0520 con 0x7f9b781012a0 2023-07-20T17:48:07.313 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7d304700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b78100900 0x7f9b78101ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:07.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7d304700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b78100900 0x7f9b78101ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36082/0 (socket says 172.21.15.49:36082) 2023-07-20T17:48:07.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7d304700 1 -- 172.21.15.49:0/3431293900 learned_addr learned my addr 172.21.15.49:0/3431293900 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:07.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7e306700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b781012a0 0x7f9b78102220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:07.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7db05700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b780ff700 0x7f9b78103690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:07.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7d304700 1 -- 172.21.15.49:0/3431293900 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b781012a0 msgr2=0x7f9b78102220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:07.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7d304700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b781012a0 0x7f9b78102220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:07.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7d304700 1 -- 172.21.15.49:0/3431293900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b780ff700 msgr2=0x7f9b78103690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:07.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7d304700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b780ff700 0x7f9b78103690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:07.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7d304700 1 -- 172.21.15.49:0/3431293900 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c0095f0 con 0x7f9b78100900 2023-07-20T17:48:07.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7d304700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b78100900 0x7f9b78101ce0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009990 tx=0x7f9b6c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:07.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b6affd700 1 -- 172.21.15.49:0/3431293900 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c01d070 con 0x7f9b78100900 2023-07-20T17:48:07.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b6affd700 1 -- 172.21.15.49:0/3431293900 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b6c005030 con 0x7f9b78100900 2023-07-20T17:48:07.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b7fd69700 1 -- 172.21.15.49:0/3431293900 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b781a07a0 con 0x7f9b78100900 2023-07-20T17:48:07.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.307+0000 7f9b6affd700 1 -- 172.21.15.49:0/3431293900 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c004b30 con 0x7f9b78100900 2023-07-20T17:48:07.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.308+0000 7f9b7fd69700 1 -- 172.21.15.49:0/3431293900 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b781a0c10 con 0x7f9b78100900 2023-07-20T17:48:07.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.308+0000 7f9b7fd69700 1 -- 172.21.15.49:0/3431293900 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7810af60 con 0x7f9b78100900 2023-07-20T17:48:07.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.310+0000 7f9b6affd700 1 -- 172.21.15.49:0/3431293900 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9b6c005410 con 0x7f9b78100900 2023-07-20T17:48:07.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.310+0000 7f9b6affd700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9b640a11d0 0x7f9b640a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:07.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.310+0000 7f9b7db05700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9b640a11d0 0x7f9b640a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:07.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.310+0000 7f9b6affd700 1 -- 172.21.15.49:0/3431293900 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f9b6c0cd500 con 0x7f9b78100900 2023-07-20T17:48:07.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.311+0000 7f9b7db05700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9b640a11d0 0x7f9b640a3690 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9b78100760 tx=0x7f9b74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:07.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.314+0000 7f9b6affd700 1 -- 172.21.15.49:0/3431293900 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9b6c0972b0 con 0x7f9b78100900 2023-07-20T17:48:07.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:07 smithi049 ceph-mon[96530]: from='osd.11 [v2:172.21.15.97:6824/1087893461,v1:172.21.15.97:6825/1087893461]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T17:48:07.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:07 smithi049 ceph-mon[96530]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T17:48:07.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.510+0000 7f9b7fd69700 1 -- 172.21.15.49:0/3431293900 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f9b781a1250 con 0x7f9b78100900 2023-07-20T17:48:07.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.511+0000 7f9b6affd700 1 -- 172.21.15.49:0/3431293900 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) v1 ==== 74+0+133 (secure 0 0 0) 0x7f9b6c09a8d0 con 0x7f9b78100900 2023-07-20T17:48:07.513 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:07.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 -- 172.21.15.49:0/3431293900 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9b640a11d0 msgr2=0x7f9b640a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:07.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9b640a11d0 0x7f9b640a3690 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9b78100760 tx=0x7f9b74009260 comp rx=0 tx=0).stop 2023-07-20T17:48:07.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 -- 172.21.15.49:0/3431293900 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b78100900 msgr2=0x7f9b78101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:07.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b78100900 0x7f9b78101ce0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009990 tx=0x7f9b6c00f690 comp rx=0 tx=0).stop 2023-07-20T17:48:07.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 -- 172.21.15.49:0/3431293900 shutdown_connections 2023-07-20T17:48:07.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f9b640a11d0 0x7f9b640a3690 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:07.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b781012a0 0x7f9b78102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:07.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b780ff700 0x7f9b78103690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:07.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 --2- 172.21.15.49:0/3431293900 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b78100900 0x7f9b78101ce0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:07.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 -- 172.21.15.49:0/3431293900 >> 172.21.15.49:0/3431293900 conn(0x7f9b780fac80 msgr2=0x7f9b780fb530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:07.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 -- 172.21.15.49:0/3431293900 shutdown_connections 2023-07-20T17:48:07.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:07.513+0000 7f9b7fd69700 1 -- 172.21.15.49:0/3431293900 wait complete. 2023-07-20T17:48:07.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:07 smithi079 ceph-mon[98036]: from='osd.11 [v2:172.21.15.97:6824/1087893461,v1:172.21.15.97:6825/1087893461]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T17:48:07.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:07 smithi079 ceph-mon[98036]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T17:48:07.909 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":75,"num_osds":12,"num_up_osds":11,"osd_up_since":1689875267,"num_in_osds":12,"osd_in_since":1689875271,"num_remapped_pgs":0} 2023-07-20T17:48:08.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:48:08.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T17:48:08.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T17:48:08.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:08.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='osd.11 [v2:172.21.15.97:6824/1087893461,v1:172.21.15.97:6825/1087893461]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:48:08.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:48:08.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3431293900' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T17:48:08.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:08.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:08.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:08 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:08.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:48:08.223+0000 7fc5b5317700 -1 osd.11 0 waiting for initial osdmap 2023-07-20T17:48:08.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:48:08.235+0000 7fc5ad105700 -1 osd.11 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T17:48:08.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:48:08.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T17:48:08.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T17:48:08.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:08.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='osd.11 [v2:172.21.15.97:6824/1087893461,v1:172.21.15.97:6825/1087893461]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:48:08.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:48:08.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3431293900' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T17:48:08.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:08.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:08.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:08 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:08.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:48:08.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T17:48:08.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T17:48:08.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:08.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='osd.11 [v2:172.21.15.97:6824/1087893461,v1:172.21.15.97:6825/1087893461]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:48:08.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-07-20T17:48:08.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3431293900' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T17:48:08.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T17:48:08.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:08.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:08.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:08.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:08 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:08.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T17:48:09.473 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:48:09.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:09 smithi049 ceph-mon[96530]: Detected new or changed devices on smithi097 2023-07-20T17:48:09.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:09 smithi049 ceph-mon[96530]: Adjusting osd_memory_target on smithi097 to 4292M 2023-07-20T17:48:09.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:09 smithi049 ceph-mon[96530]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:48:09.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:09 smithi049 ceph-mon[96530]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T17:48:09.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:09 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:09.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:09 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:09.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:09 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:09.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:09 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:09.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:09 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:09.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:09 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:09.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:09 smithi097 ceph-mon[98398]: Detected new or changed devices on smithi097 2023-07-20T17:48:09.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:09 smithi097 ceph-mon[98398]: Adjusting osd_memory_target on smithi097 to 4292M 2023-07-20T17:48:09.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:09 smithi097 ceph-mon[98398]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:48:09.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:09 smithi097 ceph-mon[98398]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T17:48:09.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:09 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:09.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:09 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:09.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:09 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:09.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:09 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:09.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:09 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:09.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:09 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:09.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:09 smithi079 ceph-mon[98036]: Detected new or changed devices on smithi097 2023-07-20T17:48:09.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:09 smithi079 ceph-mon[98036]: Adjusting osd_memory_target on smithi097 to 4292M 2023-07-20T17:48:09.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:09 smithi079 ceph-mon[98036]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-07-20T17:48:09.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:09 smithi079 ceph-mon[98036]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T17:48:09.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:09 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:09.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:09 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:09.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:09 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:09.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:09 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:09.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:09 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:09.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:09 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:10.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:10 smithi049 ceph-mon[96530]: purged_snaps scrub starts 2023-07-20T17:48:10.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:10 smithi049 ceph-mon[96530]: purged_snaps scrub ok 2023-07-20T17:48:10.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:10 smithi049 ceph-mon[96530]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:48:10.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:10 smithi049 ceph-mon[96530]: osd.11 [v2:172.21.15.97:6824/1087893461,v1:172.21.15.97:6825/1087893461] boot 2023-07-20T17:48:10.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:10 smithi049 ceph-mon[96530]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T17:48:10.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:10 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:10.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:10 smithi097 ceph-mon[98398]: purged_snaps scrub starts 2023-07-20T17:48:10.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:10 smithi097 ceph-mon[98398]: purged_snaps scrub ok 2023-07-20T17:48:10.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:10 smithi097 ceph-mon[98398]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:48:10.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:10 smithi097 ceph-mon[98398]: osd.11 [v2:172.21.15.97:6824/1087893461,v1:172.21.15.97:6825/1087893461] boot 2023-07-20T17:48:10.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:10 smithi097 ceph-mon[98398]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T17:48:10.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:10 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:10.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:10 smithi079 ceph-mon[98036]: purged_snaps scrub starts 2023-07-20T17:48:10.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:10 smithi079 ceph-mon[98036]: purged_snaps scrub ok 2023-07-20T17:48:10.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:10 smithi079 ceph-mon[98036]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T17:48:10.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:10 smithi079 ceph-mon[98036]: osd.11 [v2:172.21.15.97:6824/1087893461,v1:172.21.15.97:6825/1087893461] boot 2023-07-20T17:48:10.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:10 smithi079 ceph-mon[98036]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T17:48:10.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:10 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T17:48:10.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.764+0000 7ff4f3a72700 1 -- 172.21.15.49:0/930792416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ec1012a0 msgr2=0x7ff4ec107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:10.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.764+0000 7ff4f3a72700 1 --2- 172.21.15.49:0/930792416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ec1012a0 0x7ff4ec107b50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400b0d0 tx=0x7ff4e400b3e0 comp rx=0 tx=0).stop 2023-07-20T17:48:10.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.764+0000 7ff4f3a72700 1 -- 172.21.15.49:0/930792416 shutdown_connections 2023-07-20T17:48:10.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.764+0000 7ff4f3a72700 1 --2- 172.21.15.49:0/930792416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ec1012a0 0x7ff4ec107b50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:10.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.764+0000 7ff4f3a72700 1 --2- 172.21.15.49:0/930792416 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4ec0ff700 0x7ff4ec0ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:10.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.764+0000 7ff4f3a72700 1 --2- 172.21.15.49:0/930792416 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec100900 0x7ff4ec100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:10.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.764+0000 7ff4f3a72700 1 -- 172.21.15.49:0/930792416 >> 172.21.15.49:0/930792416 conn(0x7ff4ec0fac80 msgr2=0x7ff4ec0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:10.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.764+0000 7ff4f3a72700 1 -- 172.21.15.49:0/930792416 shutdown_connections 2023-07-20T17:48:10.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.764+0000 7ff4f3a72700 1 -- 172.21.15.49:0/930792416 wait complete. 2023-07-20T17:48:10.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.765+0000 7ff4f3a72700 1 Processor -- start 2023-07-20T17:48:10.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.765+0000 7ff4f3a72700 1 -- start start 2023-07-20T17:48:10.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.765+0000 7ff4f3a72700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4ec0ff700 0x7ff4ec196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:10.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.765+0000 7ff4f3a72700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ec100900 0x7ff4ec196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:10.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.765+0000 7ff4f3a72700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec1012a0 0x7ff4ec19db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:10.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.765+0000 7ff4f3a72700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec19e0e0 con 0x7ff4ec100900 2023-07-20T17:48:10.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.765+0000 7ff4f3a72700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec19e220 con 0x7ff4ec0ff700 2023-07-20T17:48:10.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.765+0000 7ff4f3a72700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec19e3b0 con 0x7ff4ec1012a0 2023-07-20T17:48:10.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.765+0000 7ff4f100d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ec100900 0x7ff4ec196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:10.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.765+0000 7ff4f100d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ec100900 0x7ff4ec196a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36098/0 (socket says 172.21.15.49:36098) 2023-07-20T17:48:10.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.765+0000 7ff4f100d700 1 -- 172.21.15.49:0/1358859106 learned_addr learned my addr 172.21.15.49:0/1358859106 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:10.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4f180e700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4ec0ff700 0x7ff4ec196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:10.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4f200f700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec1012a0 0x7ff4ec19db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:10.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4f100d700 1 -- 172.21.15.49:0/1358859106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec1012a0 msgr2=0x7ff4ec19db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:10.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4f100d700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec1012a0 0x7ff4ec19db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:10.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4f100d700 1 -- 172.21.15.49:0/1358859106 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4ec0ff700 msgr2=0x7ff4ec196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:10.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4f100d700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4ec0ff700 0x7ff4ec196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:10.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4f100d700 1 -- 172.21.15.49:0/1358859106 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e4009d00 con 0x7ff4ec100900 2023-07-20T17:48:10.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4f200f700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec1012a0 0x7ff4ec19db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T17:48:10.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4f100d700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ec100900 0x7ff4ec196a90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00c950 tx=0x7ff4dc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:10.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4e2ffd700 1 -- 172.21.15.49:0/1358859106 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc00cea0 con 0x7ff4ec100900 2023-07-20T17:48:10.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4e2ffd700 1 -- 172.21.15.49:0/1358859106 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4dc004830 con 0x7ff4ec100900 2023-07-20T17:48:10.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4f3a72700 1 -- 172.21.15.49:0/1358859106 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ec19e690 con 0x7ff4ec100900 2023-07-20T17:48:10.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4e2ffd700 1 -- 172.21.15.49:0/1358859106 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0054e0 con 0x7ff4ec100900 2023-07-20T17:48:10.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.766+0000 7ff4f3a72700 1 -- 172.21.15.49:0/1358859106 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec19eb90 con 0x7ff4ec100900 2023-07-20T17:48:10.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.767+0000 7ff4f3a72700 1 -- 172.21.15.49:0/1358859106 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ec1065f0 con 0x7ff4ec100900 2023-07-20T17:48:10.774 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.768+0000 7ff4e2ffd700 1 -- 172.21.15.49:0/1358859106 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff4dc0049a0 con 0x7ff4ec100900 2023-07-20T17:48:10.774 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.769+0000 7ff4e2ffd700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff4d80a1150 0x7ff4d80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:10.774 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.769+0000 7ff4f180e700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff4d80a1150 0x7ff4d80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:10.774 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.769+0000 7ff4e2ffd700 1 -- 172.21.15.49:0/1358859106 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff4dc0cc490 con 0x7ff4ec100900 2023-07-20T17:48:10.774 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.769+0000 7ff4f180e700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff4d80a1150 0x7ff4d80a3610 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec100760 tx=0x7ff4e800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:10.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.772+0000 7ff4e2ffd700 1 -- 172.21.15.49:0/1358859106 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff4dc096080 con 0x7ff4ec100900 2023-07-20T17:48:10.971 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.968+0000 7ff4f3a72700 1 -- 172.21.15.49:0/1358859106 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7ff4ec02cb40 con 0x7ff4ec100900 2023-07-20T17:48:10.971 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.968+0000 7ff4e2ffd700 1 -- 172.21.15.49:0/1358859106 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v78) v1 ==== 74+0+133 (secure 0 0 0) 0x7ff4dc01d0a0 con 0x7ff4ec100900 2023-07-20T17:48:10.972 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:10.973 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.970+0000 7ff4f3a72700 1 -- 172.21.15.49:0/1358859106 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff4d80a1150 msgr2=0x7ff4d80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:10.973 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.970+0000 7ff4f3a72700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff4d80a1150 0x7ff4d80a3610 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec100760 tx=0x7ff4e800b410 comp rx=0 tx=0).stop 2023-07-20T17:48:10.974 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.970+0000 7ff4f3a72700 1 -- 172.21.15.49:0/1358859106 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ec100900 msgr2=0x7ff4ec196a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:10.974 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.970+0000 7ff4f3a72700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ec100900 0x7ff4ec196a90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00c950 tx=0x7ff4dc00cc60 comp rx=0 tx=0).stop 2023-07-20T17:48:10.974 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.970+0000 7ff4f3a72700 1 -- 172.21.15.49:0/1358859106 shutdown_connections 2023-07-20T17:48:10.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.970+0000 7ff4f3a72700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff4d80a1150 0x7ff4d80a3610 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:10.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.970+0000 7ff4f3a72700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec1012a0 0x7ff4ec19db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:10.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.970+0000 7ff4f3a72700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4ec0ff700 0x7ff4ec196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:10.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.970+0000 7ff4f3a72700 1 --2- 172.21.15.49:0/1358859106 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ec100900 0x7ff4ec196a90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:10.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.970+0000 7ff4f3a72700 1 -- 172.21.15.49:0/1358859106 >> 172.21.15.49:0/1358859106 conn(0x7ff4ec0fac80 msgr2=0x7ff4ec0fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:10.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.971+0000 7ff4f3a72700 1 -- 172.21.15.49:0/1358859106 shutdown_connections 2023-07-20T17:48:10.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:10.971+0000 7ff4f3a72700 1 -- 172.21.15.49:0/1358859106 wait complete. 2023-07-20T17:48:11.302 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:11 smithi049 ceph-mon[96530]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T17:48:11.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:11 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1358859106' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T17:48:11.333 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":78,"num_osds":12,"num_up_osds":12,"osd_up_since":1689875289,"num_in_osds":12,"osd_in_since":1689875271,"num_remapped_pgs":0} 2023-07-20T17:48:11.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph osd dump --format=json 2023-07-20T17:48:11.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:11 smithi097 ceph-mon[98398]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T17:48:11.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:11 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1358859106' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T17:48:11.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:11 smithi079 ceph-mon[98036]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T17:48:11.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:11 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1358859106' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T17:48:11.893 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:48:12.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:12 smithi049 ceph-mon[96530]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:12.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:12 smithi097 ceph-mon[98398]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:12.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:12 smithi079 ceph-mon[98036]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:13.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.182+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2690727189 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6a00fef20 msgr2=0x7ff6a00ff300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:13.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.182+0000 7ff6a52f7700 1 --2- 172.21.15.49:0/2690727189 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6a00fef20 0x7ff6a00ff300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff694009990 tx=0x7ff694009ca0 comp rx=0 tx=0).stop 2023-07-20T17:48:13.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.182+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2690727189 shutdown_connections 2023-07-20T17:48:13.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.182+0000 7ff6a52f7700 1 --2- 172.21.15.49:0/2690727189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6a0105ac0 0x7ff6a010d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:13.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.182+0000 7ff6a52f7700 1 --2- 172.21.15.49:0/2690727189 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6a00fef20 0x7ff6a00ff300 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:13.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.182+0000 7ff6a52f7700 1 --2- 172.21.15.49:0/2690727189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6a00ff8d0 0x7ff6a0105390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:13.186 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.182+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2690727189 >> 172.21.15.49:0/2690727189 conn(0x7ff6a0075be0 msgr2=0x7ff6a00fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:13.186 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.182+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2690727189 shutdown_connections 2023-07-20T17:48:13.186 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.182+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2690727189 wait complete. 2023-07-20T17:48:13.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff6a52f7700 1 Processor -- start 2023-07-20T17:48:13.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff6a52f7700 1 -- start start 2023-07-20T17:48:13.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff6a52f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6a00fef20 0x7ff6a019c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:13.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff6a52f7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6a00ff8d0 0x7ff6a019cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:13.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff6a52f7700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6a0105ac0 0x7ff6a0196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff6a52f7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a0196c70 con 0x7ff6a00fef20 2023-07-20T17:48:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff6a52f7700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a0196de0 con 0x7ff6a00ff8d0 2023-07-20T17:48:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff6a52f7700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a0196fa0 con 0x7ff6a0105ac0 2023-07-20T17:48:13.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff69effd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6a00fef20 0x7ff6a019c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:13.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff69effd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6a00fef20 0x7ff6a019c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36106/0 (socket says 172.21.15.49:36106) 2023-07-20T17:48:13.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff69effd700 1 -- 172.21.15.49:0/2730788572 learned_addr learned my addr 172.21.15.49:0/2730788572 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:13.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff69f7fe700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6a0105ac0 0x7ff6a0196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:13.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff69e7fc700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6a00ff8d0 0x7ff6a019cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:13.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff69effd700 1 -- 172.21.15.49:0/2730788572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6a0105ac0 msgr2=0x7ff6a0196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:13.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff69effd700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6a0105ac0 0x7ff6a0196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:13.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff69effd700 1 -- 172.21.15.49:0/2730788572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6a00ff8d0 msgr2=0x7ff6a019cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:13.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.183+0000 7ff69effd700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6a00ff8d0 0x7ff6a019cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:13.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.184+0000 7ff69effd700 1 -- 172.21.15.49:0/2730788572 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6940095f0 con 0x7ff6a00fef20 2023-07-20T17:48:13.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.184+0000 7ff69e7fc700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6a00ff8d0 0x7ff6a019cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:48:13.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.184+0000 7ff69effd700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6a00fef20 0x7ff6a019c580 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff694004760 tx=0x7ff694009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:13.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.184+0000 7ff687fff700 1 -- 172.21.15.49:0/2730788572 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69401d070 con 0x7ff6a00fef20 2023-07-20T17:48:13.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.184+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2730788572 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a0197220 con 0x7ff6a00fef20 2023-07-20T17:48:13.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.184+0000 7ff687fff700 1 -- 172.21.15.49:0/2730788572 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff694004cd0 con 0x7ff6a00fef20 2023-07-20T17:48:13.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.184+0000 7ff687fff700 1 -- 172.21.15.49:0/2730788572 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff694005290 con 0x7ff6a00fef20 2023-07-20T17:48:13.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.184+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2730788572 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a01a3570 con 0x7ff6a00fef20 2023-07-20T17:48:13.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.185+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2730788572 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a004e470 con 0x7ff6a00fef20 2023-07-20T17:48:13.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.186+0000 7ff687fff700 1 -- 172.21.15.49:0/2730788572 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff6940053f0 con 0x7ff6a00fef20 2023-07-20T17:48:13.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.186+0000 7ff687fff700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff6880a1150 0x7ff6880a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:13.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.187+0000 7ff687fff700 1 -- 172.21.15.49:0/2730788572 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff6940cd3c0 con 0x7ff6a00fef20 2023-07-20T17:48:13.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.187+0000 7ff69e7fc700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff6880a1150 0x7ff6880a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:13.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.187+0000 7ff69e7fc700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff6880a1150 0x7ff6880a3610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff690005950 tx=0x7ff6900058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:13.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.191+0000 7ff687fff700 1 -- 172.21.15.49:0/2730788572 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff694096fa0 con 0x7ff6a00fef20 2023-07-20T17:48:13.394 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.391+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2730788572 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6a0197b10 con 0x7ff6a00fef20 2023-07-20T17:48:13.395 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.392+0000 7ff687fff700 1 -- 172.21.15.49:0/2730788572 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18314 (secure 0 0 0) 0x7ff69409a5c0 con 0x7ff6a00fef20 2023-07-20T17:48:13.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:13.396 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":78,"fsid":"8c050cea-2724-11ee-9b34-001a4aab830c","created":"2023-07-20T17:41:57.467323+0000","modified":"2023-07-20T17:48:10.226553+0000","last_up_change":"2023-07-20T17:48:09.220884+0000","last_in_change":"2023-07-20T17:47:51.156799+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T17:45:01.297180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63bfce41-a0a8-49b6-aa1e-ed7809c0627e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6803","nonce":2306317776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6805","nonce":2306317776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6809","nonce":2306317776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6807","nonce":2306317776}]},"public_addr":"172.21.15.49:6803/2306317776","cluster_addr":"172.21.15.49:6805/2306317776","heartbeat_back_addr":"172.21.15.49:6809/2306317776","heartbeat_front_addr":"172.21.15.49:6807/2306317776","state":["exists","up"]},{"osd":1,"uuid":"48cd9d65-b37d-410a-8007-f53fbd24dd73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6801","nonce":2048064928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6803","nonce":2048064928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6807","nonce":2048064928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6805","nonce":2048064928}]},"public_addr":"172.21.15.79:6801/2048064928","cluster_addr":"172.21.15.79:6803/2048064928","heartbeat_back_addr":"172.21.15.79:6807/2048064928","heartbeat_front_addr":"172.21.15.79:6805/2048064928","state":["exists","up"]},{"osd":2,"uuid":"272a17dd-e825-4990-a178-c2eb6985f37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6801","nonce":2945387274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6803","nonce":2945387274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6807","nonce":2945387274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6805","nonce":2945387274}]},"public_addr":"172.21.15.97:6801/2945387274","cluster_addr":"172.21.15.97:6803/2945387274","heartbeat_back_addr":"172.21.15.97:6807/2945387274","heartbeat_front_addr":"172.21.15.97:6805/2945387274","state":["exists","up"]},{"osd":3,"uuid":"6ba1972a-2bf4-4602-900c-11b15dcf6027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6811","nonce":178015359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6813","nonce":178015359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6817","nonce":178015359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6815","nonce":178015359}]},"public_addr":"172.21.15.49:6811/178015359","cluster_addr":"172.21.15.49:6813/178015359","heartbeat_back_addr":"172.21.15.49:6817/178015359","heartbeat_front_addr":"172.21.15.49:6815/178015359","state":["exists","up"]},{"osd":4,"uuid":"a167452b-f627-4aae-aecd-e513a5649235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6809","nonce":4037011752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6811","nonce":4037011752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6815","nonce":4037011752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6813","nonce":4037011752}]},"public_addr":"172.21.15.79:6809/4037011752","cluster_addr":"172.21.15.79:6811/4037011752","heartbeat_back_addr":"172.21.15.79:6815/4037011752","heartbeat_front_addr":"172.21.15.79:6813/4037011752","state":["exists","up"]},{"osd":5,"uuid":"def1f97c-84c4-4c87-9363-62f72142e0a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6809","nonce":2172761707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6811","nonce":2172761707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6815","nonce":2172761707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6813","nonce":2172761707}]},"public_addr":"172.21.15.97:6809/2172761707","cluster_addr":"172.21.15.97:6811/2172761707","heartbeat_back_addr":"172.21.15.97:6815/2172761707","heartbeat_front_addr":"172.21.15.97:6813/2172761707","state":["exists","up"]},{"osd":6,"uuid":"e8c200c2-0692-4d5e-a281-77c9ad6670cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6819","nonce":797242207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6821","nonce":797242207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6825","nonce":797242207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6823","nonce":797242207}]},"public_addr":"172.21.15.49:6819/797242207","cluster_addr":"172.21.15.49:6821/797242207","heartbeat_back_addr":"172.21.15.49:6825/797242207","heartbeat_front_addr":"172.21.15.49:6823/797242207","state":["exists","up"]},{"osd":7,"uuid":"847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6817","nonce":1417643292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6819","nonce":1417643292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6823","nonce":1417643292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6821","nonce":1417643292}]},"public_addr":"172.21.15.79:6817/1417643292","cluster_addr":"172.21.15.79:6819/1417643292","heartbeat_back_addr":"172.21.15.79:6823/1417643292","heartbeat_front_addr":"172.21.15.79:6821/1417643292","state":["exists","up"]},{"osd":8,"uuid":"14c55ac6-0e96-42a4-b974-178c87c2185c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6817","nonce":1465924719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6819","nonce":1465924719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6823","nonce":1465924719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6821","nonce":1465924719}]},"public_addr":"172.21.15.97:6817/1465924719","cluster_addr":"172.21.15.97:6819/1465924719","heartbeat_back_addr":"172.21.15.97:6823/1465924719","heartbeat_front_addr":"172.21.15.97:6821/1465924719","state":["exists","up"]},{"osd":9,"uuid":"b0c07e94-8039-4e41-85b7-23344124687b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6827","nonce":845176131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6829","nonce":845176131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6833","nonce":845176131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6831","nonce":845176131}]},"public_addr":"172.21.15.49:6827/845176131","cluster_addr":"172.21.15.49:6829/845176131","heartbeat_back_addr":"172.21.15.49:6833/845176131","heartbeat_front_addr":"172.21.15.49:6831/845176131","state":["exists","up"]},{"osd":10,"uuid":"67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6825","nonce":935518328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6827","nonce":935518328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6831","nonce":935518328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6829","nonce":935518328}]},"public_addr":"172.21.15.79:6825/935518328","cluster_addr":"172.21.15.79:6827/935518328","heartbeat_back_addr":"172.21.15.79:6831/935518328","heartbeat_front_addr":"172.21.15.79:6829/935518328","state":["exists","up"]},{"osd":11,"uuid":"c2ad9939-51b9-481e-b454-8cfc7451065b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6825","nonce":1087893461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6827","nonce":1087893461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6831","nonce":1087893461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6829","nonce":1087893461}]},"public_addr":"172.21.15.97:6825/1087893461","cluster_addr":"172.21.15.97:6827/1087893461","heartbeat_back_addr":"172.21.15.97:6831/1087893461","heartbeat_front_addr":"172.21.15.97:6829/1087893461","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:16.085186+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:37.523233+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:58.760655+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:45:20.737876+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:45:41.148757+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:02.139676+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:23.295222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:43.460985+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:04.429151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:25.761472+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:46.048313+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:48:07.381968+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3795991287":"2023-07-21T17:42:52.746152+0000","172.21.15.49:0/3777764053":"2023-07-21T17:42:52.746152+0000","172.21.15.49:6801/752420140":"2023-07-21T17:42:52.746152+0000","172.21.15.49:0/2521746017":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/1021575989":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/3362614128":"2023-07-21T17:42:52.746152+0000","172.21.15.49:6801/3017374204":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/1674833015":"2023-07-21T17:42:26.351231+0000","172.21.15.49:6800/3017374204":"2023-07-21T17:42:26.351231+0000","172.21.15.49:6800/752420140":"2023-07-21T17:42:52.746152+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T17:48:13.397 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2730788572 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff6880a1150 msgr2=0x7ff6880a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:13.398 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff6880a1150 0x7ff6880a3610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff690005950 tx=0x7ff6900058e0 comp rx=0 tx=0).stop 2023-07-20T17:48:13.398 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2730788572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6a00fef20 msgr2=0x7ff6a019c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:13.398 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6a00fef20 0x7ff6a019c580 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff694004760 tx=0x7ff694009580 comp rx=0 tx=0).stop 2023-07-20T17:48:13.398 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2730788572 shutdown_connections 2023-07-20T17:48:13.399 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff6880a1150 0x7ff6880a3610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:13.399 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6a0105ac0 0x7ff6a0196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:13.399 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6a00fef20 0x7ff6a019c580 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:13.399 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 --2- 172.21.15.49:0/2730788572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6a00ff8d0 0x7ff6a019cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:13.400 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2730788572 >> 172.21.15.49:0/2730788572 conn(0x7ff6a0075be0 msgr2=0x7ff6a01042d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:13.400 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2730788572 shutdown_connections 2023-07-20T17:48:13.400 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:13.395+0000 7ff6a52f7700 1 -- 172.21.15.49:0/2730788572 wait complete. 2023-07-20T17:48:13.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:13 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2730788572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:13.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:13 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/2730788572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:13.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:13 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/2730788572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:13.858 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T17:45:01.297180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T17:48:13.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-20T17:48:14.418 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T17:48:14.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:14 smithi049 ceph-mon[96530]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:14.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:14 smithi097 ceph-mon[98398]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:14.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:14 smithi079 ceph-mon[98036]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:15.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.885+0000 7fd8f74f6700 1 -- 172.21.15.49:0/638193836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f0104580 msgr2=0x7fd8f0104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:15.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.885+0000 7fd8f74f6700 1 --2- 172.21.15.49:0/638193836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f0104580 0x7fd8f0104960 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009990 tx=0x7fd8e0009ca0 comp rx=0 tx=0).stop 2023-07-20T17:48:15.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.885+0000 7fd8f74f6700 1 -- 172.21.15.49:0/638193836 shutdown_connections 2023-07-20T17:48:15.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.885+0000 7fd8f74f6700 1 --2- 172.21.15.49:0/638193836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8f0100070 0x7fd8f010c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:15.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.885+0000 7fd8f74f6700 1 --2- 172.21.15.49:0/638193836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f0104580 0x7fd8f0104960 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:15.889 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.885+0000 7fd8f74f6700 1 --2- 172.21.15.49:0/638193836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f00ff6b0 0x7fd8f00ffb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:15.889 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.885+0000 7fd8f74f6700 1 -- 172.21.15.49:0/638193836 >> 172.21.15.49:0/638193836 conn(0x7fd8f0076770 msgr2=0x7fd8f0076b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:15.889 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.885+0000 7fd8f74f6700 1 -- 172.21.15.49:0/638193836 shutdown_connections 2023-07-20T17:48:15.889 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.885+0000 7fd8f74f6700 1 -- 172.21.15.49:0/638193836 wait complete. 2023-07-20T17:48:15.890 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f74f6700 1 Processor -- start 2023-07-20T17:48:15.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f74f6700 1 -- start start 2023-07-20T17:48:15.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f74f6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f00ff6b0 0x7fd8f0197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:15.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f74f6700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f0100070 0x7fd8f0197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:15.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f74f6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8f0104580 0x7fd8f01a27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:15.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f74f6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f01985a0 con 0x7fd8f00ff6b0 2023-07-20T17:48:15.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f74f6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f01a2ce0 con 0x7fd8f0104580 2023-07-20T17:48:15.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f74f6700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f01a2ea0 con 0x7fd8f0100070 2023-07-20T17:48:15.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f5292700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f00ff6b0 0x7fd8f0197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:15.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f5292700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f00ff6b0 0x7fd8f0197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36124/0 (socket says 172.21.15.49:36124) 2023-07-20T17:48:15.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f5292700 1 -- 172.21.15.49:0/3063269616 learned_addr learned my addr 172.21.15.49:0/3063269616 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:15.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f4a91700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f0100070 0x7fd8f0197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:15.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f5a93700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8f0104580 0x7fd8f01a27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:15.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f5292700 1 -- 172.21.15.49:0/3063269616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f0100070 msgr2=0x7fd8f0197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:15.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f5292700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f0100070 0x7fd8f0197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:15.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f5292700 1 -- 172.21.15.49:0/3063269616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8f0104580 msgr2=0x7fd8f01a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:15.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f5292700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8f0104580 0x7fd8f01a27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:15.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.886+0000 7fd8f5292700 1 -- 172.21.15.49:0/3063269616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e00095f0 con 0x7fd8f00ff6b0 2023-07-20T17:48:15.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.887+0000 7fd8f5292700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f00ff6b0 0x7fd8f0197720 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00095c0 tx=0x7fd8e000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:15.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.887+0000 7fd8e67fc700 1 -- 172.21.15.49:0/3063269616 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e001d070 con 0x7fd8f00ff6b0 2023-07-20T17:48:15.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.887+0000 7fd8e67fc700 1 -- 172.21.15.49:0/3063269616 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8e000fb50 con 0x7fd8f00ff6b0 2023-07-20T17:48:15.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.887+0000 7fd8f74f6700 1 -- 172.21.15.49:0/3063269616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f01a3120 con 0x7fd8f00ff6b0 2023-07-20T17:48:15.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.887+0000 7fd8e67fc700 1 -- 172.21.15.49:0/3063269616 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e00173f0 con 0x7fd8f00ff6b0 2023-07-20T17:48:15.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.887+0000 7fd8f74f6700 1 -- 172.21.15.49:0/3063269616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f01a35c0 con 0x7fd8f00ff6b0 2023-07-20T17:48:15.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.888+0000 7fd8f74f6700 1 -- 172.21.15.49:0/3063269616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f004e470 con 0x7fd8f00ff6b0 2023-07-20T17:48:15.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.889+0000 7fd8e67fc700 1 -- 172.21.15.49:0/3063269616 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd8e000fcc0 con 0x7fd8f00ff6b0 2023-07-20T17:48:15.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.889+0000 7fd8e67fc700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd8dc0a1150 0x7fd8dc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:15.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.890+0000 7fd8e67fc700 1 -- 172.21.15.49:0/3063269616 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd8e00cd580 con 0x7fd8f00ff6b0 2023-07-20T17:48:15.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.890+0000 7fd8f4a91700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd8dc0a1150 0x7fd8dc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:15.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.890+0000 7fd8f4a91700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd8dc0a1150 0x7fd8dc0a3610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0068b00 tx=0x7fd8ec00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:15.896 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:15.894+0000 7fd8e67fc700 1 -- 172.21.15.49:0/3063269616 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd8e005c780 con 0x7fd8f00ff6b0 2023-07-20T17:48:16.093 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.091+0000 7fd8f74f6700 1 -- 172.21.15.49:0/3063269616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd8f01a37e0 con 0x7fd8f00ff6b0 2023-07-20T17:48:16.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.091+0000 7fd8e67fc700 1 -- 172.21.15.49:0/3063269616 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd8e0097200 con 0x7fd8f00ff6b0 2023-07-20T17:48:16.094 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2023-07-20T17:48:16.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 -- 172.21.15.49:0/3063269616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd8dc0a1150 msgr2=0x7fd8dc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:16.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd8dc0a1150 0x7fd8dc0a3610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0068b00 tx=0x7fd8ec00a300 comp rx=0 tx=0).stop 2023-07-20T17:48:16.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 -- 172.21.15.49:0/3063269616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f00ff6b0 msgr2=0x7fd8f0197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:16.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f00ff6b0 0x7fd8f0197720 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00095c0 tx=0x7fd8e000f710 comp rx=0 tx=0).stop 2023-07-20T17:48:16.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 -- 172.21.15.49:0/3063269616 shutdown_connections 2023-07-20T17:48:16.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd8dc0a1150 0x7fd8dc0a3610 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:16.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8f0104580 0x7fd8f01a27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:16.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f00ff6b0 0x7fd8f0197720 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:16.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 --2- 172.21.15.49:0/3063269616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd8f0100070 0x7fd8f0197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:16.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 -- 172.21.15.49:0/3063269616 >> 172.21.15.49:0/3063269616 conn(0x7fd8f0076770 msgr2=0x7fd8f00fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:16.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 -- 172.21.15.49:0/3063269616 shutdown_connections 2023-07-20T17:48:16.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:16.093+0000 7fd8f74f6700 1 -- 172.21.15.49:0/3063269616 wait complete. 2023-07-20T17:48:16.476 INFO:tasks.cephadm:Adding mds.a on smithi049 2023-07-20T17:48:16.476 INFO:tasks.cephadm:Adding mds.d on smithi049 2023-07-20T17:48:16.477 INFO:tasks.cephadm:Adding mds.g on smithi049 2023-07-20T17:48:16.477 INFO:tasks.cephadm:Adding mds.j on smithi049 2023-07-20T17:48:16.477 INFO:tasks.cephadm:Adding mds.b on smithi079 2023-07-20T17:48:16.477 INFO:tasks.cephadm:Adding mds.e on smithi079 2023-07-20T17:48:16.477 INFO:tasks.cephadm:Adding mds.h on smithi079 2023-07-20T17:48:16.478 INFO:tasks.cephadm:Adding mds.k on smithi079 2023-07-20T17:48:16.478 INFO:tasks.cephadm:Adding mds.c on smithi097 2023-07-20T17:48:16.478 INFO:tasks.cephadm:Adding mds.f on smithi097 2023-07-20T17:48:16.478 INFO:tasks.cephadm:Adding mds.i on smithi097 2023-07-20T17:48:16.478 INFO:tasks.cephadm:Adding mds.l on smithi097 2023-07-20T17:48:16.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c050cea-2724-11ee-9b34-001a4aab830c -- ceph orch apply mds all '12;smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi097=c;smithi097=f;smithi097=i;smithi097=l' 2023-07-20T17:48:16.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:16 smithi049 ceph-mon[96530]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:16.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:16 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3063269616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T17:48:16.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:16 smithi097 ceph-mon[98398]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:16.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:16 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3063269616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T17:48:16.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:16 smithi079 ceph-mon[98036]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:16.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:16 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3063269616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T17:48:17.047 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T17:48:18.272 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.269+0000 7fc9f61e5700 1 -- 172.21.15.97:0/2966113407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f00ff700 msgr2=0x7fc9f00ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:18.272 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.269+0000 7fc9f61e5700 1 --2- 172.21.15.97:0/2966113407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f00ff700 0x7fc9f00ffb20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009990 tx=0x7fc9d8009ca0 comp rx=0 tx=0).stop 2023-07-20T17:48:18.272 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.269+0000 7fc9f61e5700 1 -- 172.21.15.97:0/2966113407 shutdown_connections 2023-07-20T17:48:18.273 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.269+0000 7fc9f61e5700 1 --2- 172.21.15.97:0/2966113407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f00ff700 0x7fc9f00ffb20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:18.273 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.269+0000 7fc9f61e5700 1 --2- 172.21.15.97:0/2966113407 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9f01012a0 0x7fc9f0107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:18.273 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.269+0000 7fc9f61e5700 1 --2- 172.21.15.97:0/2966113407 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9f0100900 0x7fc9f0100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:18.273 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.269+0000 7fc9f61e5700 1 -- 172.21.15.97:0/2966113407 >> 172.21.15.97:0/2966113407 conn(0x7fc9f00fac80 msgr2=0x7fc9f00fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:18.273 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.269+0000 7fc9f61e5700 1 -- 172.21.15.97:0/2966113407 shutdown_connections 2023-07-20T17:48:18.274 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.269+0000 7fc9f61e5700 1 -- 172.21.15.97:0/2966113407 wait complete. 2023-07-20T17:48:18.274 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.269+0000 7fc9f61e5700 1 Processor -- start 2023-07-20T17:48:18.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9f61e5700 1 -- start start 2023-07-20T17:48:18.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9f61e5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9f00ff700 0x7fc9f01964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:18.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9f61e5700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f0100900 0x7fc9f0196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:18.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9f61e5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9f01012a0 0x7fc9f019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:18.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9f61e5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f019e080 con 0x7fc9f01012a0 2023-07-20T17:48:18.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9f61e5700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f019e1c0 con 0x7fc9f00ff700 2023-07-20T17:48:18.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9f61e5700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f019e350 con 0x7fc9f0100900 2023-07-20T17:48:18.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9eeffd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f0100900 0x7fc9f0196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:18.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9eeffd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f0100900 0x7fc9f0196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:32896/0 (socket says 172.21.15.97:32896) 2023-07-20T17:48:18.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9eeffd700 1 -- 172.21.15.97:0/3947391705 learned_addr learned my addr 172.21.15.97:0/3947391705 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-07-20T17:48:18.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9ef7fe700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9f00ff700 0x7fc9f01964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:18.281 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9effff700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9f01012a0 0x7fc9f019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:18.281 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9eeffd700 1 -- 172.21.15.97:0/3947391705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9f00ff700 msgr2=0x7fc9f01964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:18.281 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9eeffd700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9f00ff700 0x7fc9f01964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:18.281 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9eeffd700 1 -- 172.21.15.97:0/3947391705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9f01012a0 msgr2=0x7fc9f019dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:18.282 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9eeffd700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9f01012a0 0x7fc9f019dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:18.282 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9eeffd700 1 -- 172.21.15.97:0/3947391705 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d80095f0 con 0x7fc9f0100900 2023-07-20T17:48:18.282 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9eeffd700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f0100900 0x7fc9f0196a30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000d980 tx=0x7fc9e000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:18.282 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9ecff9700 1 -- 172.21.15.97:0/3947391705 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0009b40 con 0x7fc9f0100900 2023-07-20T17:48:18.282 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.270+0000 7fc9ecff9700 1 -- 172.21.15.97:0/3947391705 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9e000b6e0 con 0x7fc9f0100900 2023-07-20T17:48:18.283 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.271+0000 7fc9f61e5700 1 -- 172.21.15.97:0/3947391705 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f019e630 con 0x7fc9f0100900 2023-07-20T17:48:18.283 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.271+0000 7fc9ecff9700 1 -- 172.21.15.97:0/3947391705 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e000f620 con 0x7fc9f0100900 2023-07-20T17:48:18.283 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.271+0000 7fc9f61e5700 1 -- 172.21.15.97:0/3947391705 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f019eb30 con 0x7fc9f0100900 2023-07-20T17:48:18.284 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.272+0000 7fc9f61e5700 1 -- 172.21.15.97:0/3947391705 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f0106690 con 0x7fc9f0100900 2023-07-20T17:48:18.284 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.273+0000 7fc9ecff9700 1 -- 172.21.15.97:0/3947391705 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc9e0010460 con 0x7fc9f0100900 2023-07-20T17:48:18.284 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.273+0000 7fc9ecff9700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9dc0a1150 0x7fc9dc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:18.284 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.274+0000 7fc9ecff9700 1 -- 172.21.15.97:0/3947391705 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc9e00cc6f0 con 0x7fc9f0100900 2023-07-20T17:48:18.284 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.274+0000 7fc9ef7fe700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9dc0a1150 0x7fc9dc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:18.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.274+0000 7fc9ef7fe700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9dc0a1150 0x7fc9dc0a3610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8000c00 tx=0x7fc9d8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:18.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.277+0000 7fc9ecff9700 1 -- 172.21.15.97:0/3947391705 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc9e00962e0 con 0x7fc9f0100900 2023-07-20T17:48:18.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.472+0000 7fc9f61e5700 1 -- 172.21.15.97:0/3947391705 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi097=c;smithi097=f;smithi097=i;smithi097=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9f0060d70 con 0x7fc9dc0a1150 2023-07-20T17:48:18.483 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.481+0000 7fc9ecff9700 1 -- 172.21.15.97:0/3947391705 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fc9f0060d70 con 0x7fc9dc0a1150 2023-07-20T17:48:18.484 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled mds.all update... 2023-07-20T17:48:18.486 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 -- 172.21.15.97:0/3947391705 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9dc0a1150 msgr2=0x7fc9dc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:18.486 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9dc0a1150 0x7fc9dc0a3610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8000c00 tx=0x7fc9d8005fb0 comp rx=0 tx=0).stop 2023-07-20T17:48:18.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 -- 172.21.15.97:0/3947391705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f0100900 msgr2=0x7fc9f0196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:18.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f0100900 0x7fc9f0196a30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000d980 tx=0x7fc9e000dc90 comp rx=0 tx=0).stop 2023-07-20T17:48:18.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 -- 172.21.15.97:0/3947391705 shutdown_connections 2023-07-20T17:48:18.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fc9dc0a1150 0x7fc9dc0a3610 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:18.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9f01012a0 0x7fc9f019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:18.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc9f00ff700 0x7fc9f01964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:18.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 --2- 172.21.15.97:0/3947391705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc9f0100900 0x7fc9f0196a30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:18.488 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 -- 172.21.15.97:0/3947391705 >> 172.21.15.97:0/3947391705 conn(0x7fc9f00fac80 msgr2=0x7fc9f00fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:18.488 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 -- 172.21.15.97:0/3947391705 shutdown_connections 2023-07-20T17:48:18.488 INFO:teuthology.orchestra.run.smithi097.stderr:2023-07-20T17:48:18.483+0000 7fc9f61e5700 1 -- 172.21.15.97:0/3947391705 wait complete. 2023-07-20T17:48:18.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:18 smithi049 ceph-mon[96530]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:18.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:18 smithi097 ceph-mon[98398]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:18.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:18 smithi079 ceph-mon[98036]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:18.901 DEBUG:teuthology.orchestra.run.smithi049:mds.a> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.a.service 2023-07-20T17:48:18.904 DEBUG:teuthology.orchestra.run.smithi049:mds.d> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.d.service 2023-07-20T17:48:18.942 INFO:journalctl@ceph.mds.a.smithi049.stdout:-- Logs begin at Thu 2023-07-20 17:16:49 UTC. -- 2023-07-20T17:48:18.943 DEBUG:teuthology.orchestra.run.smithi049:mds.g> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.g.service 2023-07-20T17:48:18.974 INFO:journalctl@ceph.mds.d.smithi049.stdout:-- Logs begin at Thu 2023-07-20 17:16:49 UTC. -- 2023-07-20T17:48:18.975 DEBUG:teuthology.orchestra.run.smithi049:mds.j> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.j.service 2023-07-20T17:48:19.003 INFO:journalctl@ceph.mds.g.smithi049.stdout:-- Logs begin at Thu 2023-07-20 17:16:49 UTC. -- 2023-07-20T17:48:19.004 DEBUG:teuthology.orchestra.run.smithi079:mds.b> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.b.service 2023-07-20T17:48:19.007 DEBUG:teuthology.orchestra.run.smithi079:mds.e> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.e.service 2023-07-20T17:48:19.032 INFO:journalctl@ceph.mds.j.smithi049.stdout:-- Logs begin at Thu 2023-07-20 17:16:49 UTC. -- 2023-07-20T17:48:19.044 INFO:journalctl@ceph.mds.b.smithi079.stdout:-- Logs begin at Thu 2023-07-20 17:15:42 UTC. -- 2023-07-20T17:48:19.045 DEBUG:teuthology.orchestra.run.smithi079:mds.h> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.h.service 2023-07-20T17:48:19.074 INFO:journalctl@ceph.mds.e.smithi079.stdout:-- Logs begin at Thu 2023-07-20 17:15:42 UTC. -- 2023-07-20T17:48:19.075 DEBUG:teuthology.orchestra.run.smithi079:mds.k> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.k.service 2023-07-20T17:48:19.103 INFO:journalctl@ceph.mds.h.smithi079.stdout:-- Logs begin at Thu 2023-07-20 17:15:42 UTC. -- 2023-07-20T17:48:19.104 DEBUG:teuthology.orchestra.run.smithi097:mds.c> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.c.service 2023-07-20T17:48:19.106 DEBUG:teuthology.orchestra.run.smithi097:mds.f> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.f.service 2023-07-20T17:48:19.130 INFO:journalctl@ceph.mds.k.smithi079.stdout:-- Logs begin at Thu 2023-07-20 17:15:42 UTC. -- 2023-07-20T17:48:19.136 INFO:journalctl@ceph.mds.c.smithi097.stdout:-- Logs begin at Thu 2023-07-20 17:15:39 UTC. -- 2023-07-20T17:48:19.137 DEBUG:teuthology.orchestra.run.smithi097:mds.i> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.i.service 2023-07-20T17:48:19.164 INFO:journalctl@ceph.mds.f.smithi097.stdout:-- Logs begin at Thu 2023-07-20 17:15:39 UTC. -- 2023-07-20T17:48:19.166 DEBUG:teuthology.orchestra.run.smithi097:mds.l> sudo journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.l.service 2023-07-20T17:48:19.191 INFO:journalctl@ceph.mds.i.smithi097.stdout:-- Logs begin at Thu 2023-07-20 17:15:39 UTC. -- 2023-07-20T17:48:19.192 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-20T17:48:19.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T17:48:19.221 INFO:journalctl@ceph.mds.l.smithi097.stdout:-- Logs begin at Thu 2023-07-20 17:15:39 UTC. -- 2023-07-20T17:48:19.338 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.333+0000 7f5876b2a700 1 -- 172.21.15.49:0/290064072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5870100910 msgr2=0x7f5870100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:19.338 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.333+0000 7f5876b2a700 1 --2- 172.21.15.49:0/290064072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5870100910 0x7f5870100d70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f585c0099c0 tx=0x7f585c009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.333+0000 7f5876b2a700 1 -- 172.21.15.49:0/290064072 shutdown_connections 2023-07-20T17:48:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.333+0000 7f5876b2a700 1 --2- 172.21.15.49:0/290064072 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58701012b0 0x7f5870107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.333+0000 7f5876b2a700 1 --2- 172.21.15.49:0/290064072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58700ff720 0x7f58700ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.333+0000 7f5876b2a700 1 --2- 172.21.15.49:0/290064072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5870100910 0x7f5870100d70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.333+0000 7f5876b2a700 1 -- 172.21.15.49:0/290064072 >> 172.21.15.49:0/290064072 conn(0x7f58700facb0 msgr2=0x7f58700fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.333+0000 7f5876b2a700 1 -- 172.21.15.49:0/290064072 shutdown_connections 2023-07-20T17:48:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.333+0000 7f5876b2a700 1 -- 172.21.15.49:0/290064072 wait complete. 2023-07-20T17:48:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f5876b2a700 1 Processor -- start 2023-07-20T17:48:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f5876b2a700 1 -- start start 2023-07-20T17:48:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f5876b2a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58700ff720 0x7f58700685b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f5876b2a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5870100910 0x7f587019b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f5876b2a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58701012b0 0x7f587019d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f5876b2a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587019e080 con 0x7f58700ff720 2023-07-20T17:48:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f5876b2a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587019e1f0 con 0x7f5870100910 2023-07-20T17:48:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f5876b2a700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587019e3b0 con 0x7f58701012b0 2023-07-20T17:48:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f58748c6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58700ff720 0x7f58700685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f58748c6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58700ff720 0x7f58700685b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36136/0 (socket says 172.21.15.49:36136) 2023-07-20T17:48:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f58748c6700 1 -- 172.21.15.49:0/529204552 learned_addr learned my addr 172.21.15.49:0/529204552 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.334+0000 7f586ffff700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5870100910 0x7f587019b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f58750c7700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58701012b0 0x7f587019d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f58748c6700 1 -- 172.21.15.49:0/529204552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58701012b0 msgr2=0x7f587019d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f58748c6700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58701012b0 0x7f587019d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f58748c6700 1 -- 172.21.15.49:0/529204552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5870100910 msgr2=0x7f587019b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f58748c6700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5870100910 0x7f587019b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f58748c6700 1 -- 172.21.15.49:0/529204552 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c0095f0 con 0x7f58700ff720 2023-07-20T17:48:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f58750c7700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58701012b0 0x7f587019d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T17:48:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f586ffff700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5870100910 0x7f587019b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T17:48:19.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f58748c6700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58700ff720 0x7f58700685b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f586400e970 tx=0x7f586400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:19.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f586dffb700 1 -- 172.21.15.49:0/529204552 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5864009790 con 0x7f58700ff720 2023-07-20T17:48:19.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f5876b2a700 1 -- 172.21.15.49:0/529204552 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587019e690 con 0x7f58700ff720 2023-07-20T17:48:19.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f586dffb700 1 -- 172.21.15.49:0/529204552 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f586400eec0 con 0x7f58700ff720 2023-07-20T17:48:19.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f586dffb700 1 -- 172.21.15.49:0/529204552 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5864003cf0 con 0x7f58700ff720 2023-07-20T17:48:19.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.335+0000 7f5876b2a700 1 -- 172.21.15.49:0/529204552 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587019ebf0 con 0x7f58700ff720 2023-07-20T17:48:19.344 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.336+0000 7f5876b2a700 1 -- 172.21.15.49:0/529204552 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58701066a0 con 0x7f58700ff720 2023-07-20T17:48:19.344 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.337+0000 7f586dffb700 1 -- 172.21.15.49:0/529204552 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f586400ccd0 con 0x7f58700ff720 2023-07-20T17:48:19.344 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.338+0000 7f586dffb700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58600a1150 0x7f58600a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:19.345 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.338+0000 7f586ffff700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58600a1150 0x7f58600a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:19.345 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.338+0000 7f586dffb700 1 -- 172.21.15.49:0/529204552 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5864014070 con 0x7f58700ff720 2023-07-20T17:48:19.345 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.338+0000 7f586ffff700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58600a1150 0x7f58600a3610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f585c00b580 tx=0x7f585c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:19.345 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.342+0000 7f586dffb700 1 -- 172.21.15.49:0/529204552 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f58640962b0 con 0x7f58700ff720 2023-07-20T17:48:19.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.543+0000 7f5876b2a700 1 -- 172.21.15.49:0/529204552 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f587002c970 con 0x7f58700ff720 2023-07-20T17:48:19.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.544+0000 7f586dffb700 1 -- 172.21.15.49:0/529204552 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18314 (secure 0 0 0) 0x7f58640998d0 con 0x7f58700ff720 2023-07-20T17:48:19.547 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:19.547 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":78,"fsid":"8c050cea-2724-11ee-9b34-001a4aab830c","created":"2023-07-20T17:41:57.467323+0000","modified":"2023-07-20T17:48:10.226553+0000","last_up_change":"2023-07-20T17:48:09.220884+0000","last_in_change":"2023-07-20T17:47:51.156799+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T17:45:01.297180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63bfce41-a0a8-49b6-aa1e-ed7809c0627e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6803","nonce":2306317776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6805","nonce":2306317776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6809","nonce":2306317776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6807","nonce":2306317776}]},"public_addr":"172.21.15.49:6803/2306317776","cluster_addr":"172.21.15.49:6805/2306317776","heartbeat_back_addr":"172.21.15.49:6809/2306317776","heartbeat_front_addr":"172.21.15.49:6807/2306317776","state":["exists","up"]},{"osd":1,"uuid":"48cd9d65-b37d-410a-8007-f53fbd24dd73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6801","nonce":2048064928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6803","nonce":2048064928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6807","nonce":2048064928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6805","nonce":2048064928}]},"public_addr":"172.21.15.79:6801/2048064928","cluster_addr":"172.21.15.79:6803/2048064928","heartbeat_back_addr":"172.21.15.79:6807/2048064928","heartbeat_front_addr":"172.21.15.79:6805/2048064928","state":["exists","up"]},{"osd":2,"uuid":"272a17dd-e825-4990-a178-c2eb6985f37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6801","nonce":2945387274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6803","nonce":2945387274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6807","nonce":2945387274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6805","nonce":2945387274}]},"public_addr":"172.21.15.97:6801/2945387274","cluster_addr":"172.21.15.97:6803/2945387274","heartbeat_back_addr":"172.21.15.97:6807/2945387274","heartbeat_front_addr":"172.21.15.97:6805/2945387274","state":["exists","up"]},{"osd":3,"uuid":"6ba1972a-2bf4-4602-900c-11b15dcf6027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6811","nonce":178015359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6813","nonce":178015359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6817","nonce":178015359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6815","nonce":178015359}]},"public_addr":"172.21.15.49:6811/178015359","cluster_addr":"172.21.15.49:6813/178015359","heartbeat_back_addr":"172.21.15.49:6817/178015359","heartbeat_front_addr":"172.21.15.49:6815/178015359","state":["exists","up"]},{"osd":4,"uuid":"a167452b-f627-4aae-aecd-e513a5649235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6809","nonce":4037011752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6811","nonce":4037011752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6815","nonce":4037011752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6813","nonce":4037011752}]},"public_addr":"172.21.15.79:6809/4037011752","cluster_addr":"172.21.15.79:6811/4037011752","heartbeat_back_addr":"172.21.15.79:6815/4037011752","heartbeat_front_addr":"172.21.15.79:6813/4037011752","state":["exists","up"]},{"osd":5,"uuid":"def1f97c-84c4-4c87-9363-62f72142e0a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6809","nonce":2172761707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6811","nonce":2172761707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6815","nonce":2172761707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6813","nonce":2172761707}]},"public_addr":"172.21.15.97:6809/2172761707","cluster_addr":"172.21.15.97:6811/2172761707","heartbeat_back_addr":"172.21.15.97:6815/2172761707","heartbeat_front_addr":"172.21.15.97:6813/2172761707","state":["exists","up"]},{"osd":6,"uuid":"e8c200c2-0692-4d5e-a281-77c9ad6670cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6819","nonce":797242207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6821","nonce":797242207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6825","nonce":797242207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6823","nonce":797242207}]},"public_addr":"172.21.15.49:6819/797242207","cluster_addr":"172.21.15.49:6821/797242207","heartbeat_back_addr":"172.21.15.49:6825/797242207","heartbeat_front_addr":"172.21.15.49:6823/797242207","state":["exists","up"]},{"osd":7,"uuid":"847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6817","nonce":1417643292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6819","nonce":1417643292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6823","nonce":1417643292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6821","nonce":1417643292}]},"public_addr":"172.21.15.79:6817/1417643292","cluster_addr":"172.21.15.79:6819/1417643292","heartbeat_back_addr":"172.21.15.79:6823/1417643292","heartbeat_front_addr":"172.21.15.79:6821/1417643292","state":["exists","up"]},{"osd":8,"uuid":"14c55ac6-0e96-42a4-b974-178c87c2185c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6817","nonce":1465924719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6819","nonce":1465924719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6823","nonce":1465924719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6821","nonce":1465924719}]},"public_addr":"172.21.15.97:6817/1465924719","cluster_addr":"172.21.15.97:6819/1465924719","heartbeat_back_addr":"172.21.15.97:6823/1465924719","heartbeat_front_addr":"172.21.15.97:6821/1465924719","state":["exists","up"]},{"osd":9,"uuid":"b0c07e94-8039-4e41-85b7-23344124687b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6827","nonce":845176131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6829","nonce":845176131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6833","nonce":845176131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6831","nonce":845176131}]},"public_addr":"172.21.15.49:6827/845176131","cluster_addr":"172.21.15.49:6829/845176131","heartbeat_back_addr":"172.21.15.49:6833/845176131","heartbeat_front_addr":"172.21.15.49:6831/845176131","state":["exists","up"]},{"osd":10,"uuid":"67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6825","nonce":935518328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6827","nonce":935518328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6831","nonce":935518328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6829","nonce":935518328}]},"public_addr":"172.21.15.79:6825/935518328","cluster_addr":"172.21.15.79:6827/935518328","heartbeat_back_addr":"172.21.15.79:6831/935518328","heartbeat_front_addr":"172.21.15.79:6829/935518328","state":["exists","up"]},{"osd":11,"uuid":"c2ad9939-51b9-481e-b454-8cfc7451065b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6825","nonce":1087893461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6827","nonce":1087893461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6831","nonce":1087893461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6829","nonce":1087893461}]},"public_addr":"172.21.15.97:6825/1087893461","cluster_addr":"172.21.15.97:6827/1087893461","heartbeat_back_addr":"172.21.15.97:6831/1087893461","heartbeat_front_addr":"172.21.15.97:6829/1087893461","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:16.085186+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:37.523233+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:58.760655+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:45:20.737876+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:45:41.148757+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:02.139676+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:23.295222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:43.460985+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:04.429151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:25.761472+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:46.048313+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:48:07.381968+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3795991287":"2023-07-21T17:42:52.746152+0000","172.21.15.49:0/3777764053":"2023-07-21T17:42:52.746152+0000","172.21.15.49:6801/752420140":"2023-07-21T17:42:52.746152+0000","172.21.15.49:0/2521746017":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/1021575989":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/3362614128":"2023-07-21T17:42:52.746152+0000","172.21.15.49:6801/3017374204":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/1674833015":"2023-07-21T17:42:26.351231+0000","172.21.15.49:6800/3017374204":"2023-07-21T17:42:26.351231+0000","172.21.15.49:6800/752420140":"2023-07-21T17:42:52.746152+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T17:48:19.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.546+0000 7f5876b2a700 1 -- 172.21.15.49:0/529204552 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58600a1150 msgr2=0x7f58600a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:19.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.546+0000 7f5876b2a700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58600a1150 0x7f58600a3610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f585c00b580 tx=0x7f585c01a040 comp rx=0 tx=0).stop 2023-07-20T17:48:19.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.546+0000 7f5876b2a700 1 -- 172.21.15.49:0/529204552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58700ff720 msgr2=0x7f58700685b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:19.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.546+0000 7f5876b2a700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58700ff720 0x7f58700685b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f586400e970 tx=0x7f586400ec80 comp rx=0 tx=0).stop 2023-07-20T17:48:19.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.546+0000 7f5876b2a700 1 -- 172.21.15.49:0/529204552 shutdown_connections 2023-07-20T17:48:19.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.546+0000 7f5876b2a700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f58600a1150 0x7f58600a3610 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.546+0000 7f5876b2a700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58701012b0 0x7f587019d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.546+0000 7f5876b2a700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f58700ff720 0x7f58700685b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.546+0000 7f5876b2a700 1 --2- 172.21.15.49:0/529204552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5870100910 0x7f587019b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.546+0000 7f5876b2a700 1 -- 172.21.15.49:0/529204552 >> 172.21.15.49:0/529204552 conn(0x7f58700facb0 msgr2=0x7f58700fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:19.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.546+0000 7f5876b2a700 1 -- 172.21.15.49:0/529204552 shutdown_connections 2023-07-20T17:48:19.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.547+0000 7f5876b2a700 1 -- 172.21.15.49:0/529204552 wait complete. 2023-07-20T17:48:19.562 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T17:45:01.297180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T17:48:19.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T17:48:19.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.689+0000 7f00c8ed2700 1 -- 172.21.15.49:0/528333627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00c4101300 msgr2=0x7f00c41036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:19.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.689+0000 7f00c8ed2700 1 --2- 172.21.15.49:0/528333627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00c4101300 0x7f00c41036f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009990 tx=0x7f00b4009ca0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.689+0000 7f00c8ed2700 1 -- 172.21.15.49:0/528333627 shutdown_connections 2023-07-20T17:48:19.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.689+0000 7f00c8ed2700 1 --2- 172.21.15.49:0/528333627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f00c4106560 0x7f00c4108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.689+0000 7f00c8ed2700 1 --2- 172.21.15.49:0/528333627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00c4101300 0x7f00c41036f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.689+0000 7f00c8ed2700 1 --2- 172.21.15.49:0/528333627 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00c4103c30 0x7f00c4106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.689+0000 7f00c8ed2700 1 -- 172.21.15.49:0/528333627 >> 172.21.15.49:0/528333627 conn(0x7f00c40facf0 msgr2=0x7f00c40fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:19.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.689+0000 7f00c8ed2700 1 -- 172.21.15.49:0/528333627 shutdown_connections 2023-07-20T17:48:19.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.689+0000 7f00c8ed2700 1 -- 172.21.15.49:0/528333627 wait complete. 2023-07-20T17:48:19.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.689+0000 7f00c8ed2700 1 Processor -- start 2023-07-20T17:48:19.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c8ed2700 1 -- start start 2023-07-20T17:48:19.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c8ed2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00c4101300 0x7f00c4196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:19.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c8ed2700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00c4103c30 0x7f00c4196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:19.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c8ed2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f00c4106560 0x7f00c419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:19.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c8ed2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c419e0c0 con 0x7f00c4101300 2023-07-20T17:48:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c8ed2700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c419e200 con 0x7f00c4106560 2023-07-20T17:48:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c8ed2700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c419e390 con 0x7f00c4103c30 2023-07-20T17:48:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00c4101300 0x7f00c4196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00c4101300 0x7f00c4196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36150/0 (socket says 172.21.15.49:36150) 2023-07-20T17:48:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c259c700 1 -- 172.21.15.49:0/1980352675 learned_addr learned my addr 172.21.15.49:0/1980352675 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c1d9b700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00c4103c30 0x7f00c4196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c2d9d700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f00c4106560 0x7f00c419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:19.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c259c700 1 -- 172.21.15.49:0/1980352675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00c4103c30 msgr2=0x7f00c4196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:19.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c259c700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00c4103c30 0x7f00c4196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c259c700 1 -- 172.21.15.49:0/1980352675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f00c4106560 msgr2=0x7f00c419daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:19.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c259c700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f00c4106560 0x7f00c419daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c259c700 1 -- 172.21.15.49:0/1980352675 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b40095f0 con 0x7f00c4101300 2023-07-20T17:48:19.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c1d9b700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00c4103c30 0x7f00c4196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:48:19.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c259c700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00c4101300 0x7f00c4196540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f00b40059d0 tx=0x7f00b400f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00b37fe700 1 -- 172.21.15.49:0/1980352675 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b401d070 con 0x7f00c4101300 2023-07-20T17:48:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00b37fe700 1 -- 172.21.15.49:0/1980352675 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00b400fd60 con 0x7f00c4101300 2023-07-20T17:48:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00c8ed2700 1 -- 172.21.15.49:0/1980352675 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c419e610 con 0x7f00c4101300 2023-07-20T17:48:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.690+0000 7f00b37fe700 1 -- 172.21.15.49:0/1980352675 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4017650 con 0x7f00c4101300 2023-07-20T17:48:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.691+0000 7f00c8ed2700 1 -- 172.21.15.49:0/1980352675 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c419ea30 con 0x7f00c4101300 2023-07-20T17:48:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.692+0000 7f00c8ed2700 1 -- 172.21.15.49:0/1980352675 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c40fc870 con 0x7f00c4101300 2023-07-20T17:48:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.693+0000 7f00b37fe700 1 -- 172.21.15.49:0/1980352675 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f00b40177b0 con 0x7f00c4101300 2023-07-20T17:48:19.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.693+0000 7f00b37fe700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f00ac0a1150 0x7f00ac0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:19.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.693+0000 7f00b37fe700 1 -- 172.21.15.49:0/1980352675 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f00b40cda30 con 0x7f00c4101300 2023-07-20T17:48:19.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.693+0000 7f00c1d9b700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f00ac0a1150 0x7f00ac0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:19.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.694+0000 7f00c1d9b700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f00ac0a1150 0x7f00ac0a3610 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f00b800bcf0 tx=0x7f00b800b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:19.700 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.698+0000 7f00b37fe700 1 -- 172.21.15.49:0/1980352675 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f00b405c780 con 0x7f00c4101300 2023-07-20T17:48:19.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:19 smithi049 ceph-mon[96530]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi097=c;smithi097=f;smithi097=i;smithi097=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:48:19.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:19 smithi049 ceph-mon[96530]: Saving service mds.all spec with placement smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi097=c;smithi097=f;smithi097=i;smithi097=l;count:12 2023-07-20T17:48:19.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:19.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:19.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:19.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:19.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:19.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:19.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:19.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:19 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:19.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:19 smithi049 ceph-mon[96530]: Deploying daemon mds.l on smithi097 2023-07-20T17:48:19.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:19 smithi097 ceph-mon[98398]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi097=c;smithi097=f;smithi097=i;smithi097=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:48:19.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:19 smithi097 ceph-mon[98398]: Saving service mds.all spec with placement smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi097=c;smithi097=f;smithi097=i;smithi097=l;count:12 2023-07-20T17:48:19.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:19 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:19.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:19 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:19.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:19 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:19.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:19 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:19.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:19 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:19.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:19 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:19.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:19 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:19.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:19 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:19.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:19 smithi097 ceph-mon[98398]: Deploying daemon mds.l on smithi097 2023-07-20T17:48:19.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:19 smithi079 ceph-mon[98036]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi097=c;smithi097=f;smithi097=i;smithi097=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T17:48:19.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:19 smithi079 ceph-mon[98036]: Saving service mds.all spec with placement smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi079=b;smithi079=e;smithi079=h;smithi079=k;smithi097=c;smithi097=f;smithi097=i;smithi097=l;count:12 2023-07-20T17:48:19.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:19 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:19.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:19 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:19.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:19 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:19.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:19 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:19.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:19 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:19.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:19 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:19.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:19 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:19.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:19 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:19.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:19 smithi079 ceph-mon[98036]: Deploying daemon mds.l on smithi097 2023-07-20T17:48:19.900 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.897+0000 7f00c8ed2700 1 -- 172.21.15.49:0/1980352675 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f00c402cbc0 con 0x7f00c4101300 2023-07-20T17:48:19.900 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.898+0000 7f00b37fe700 1 -- 172.21.15.49:0/1980352675 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f00b40976b0 con 0x7f00c4101300 2023-07-20T17:48:19.901 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2023-07-20T17:48:19.902 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 -- 172.21.15.49:0/1980352675 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f00ac0a1150 msgr2=0x7f00ac0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:19.902 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f00ac0a1150 0x7f00ac0a3610 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f00b800bcf0 tx=0x7f00b800b2c0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 -- 172.21.15.49:0/1980352675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00c4101300 msgr2=0x7f00c4196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:19.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00c4101300 0x7f00c4196540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f00b40059d0 tx=0x7f00b400f510 comp rx=0 tx=0).stop 2023-07-20T17:48:19.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 -- 172.21.15.49:0/1980352675 shutdown_connections 2023-07-20T17:48:19.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f00ac0a1150 0x7f00ac0a3610 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f00c4106560 0x7f00c419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00c4101300 0x7f00c4196540 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 --2- 172.21.15.49:0/1980352675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00c4103c30 0x7f00c4196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:19.904 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 -- 172.21.15.49:0/1980352675 >> 172.21.15.49:0/1980352675 conn(0x7f00c40facf0 msgr2=0x7f00c4101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:19.904 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 -- 172.21.15.49:0/1980352675 shutdown_connections 2023-07-20T17:48:19.904 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:19.900+0000 7f00c8ed2700 1 -- 172.21.15.49:0/1980352675 wait complete. 2023-07-20T17:48:19.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:20.044 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.038+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/1057536992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae01012f0 msgr2=0x7f8ae0107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:20.044 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.038+0000 7f8ae4d1d700 1 --2- 172.21.15.49:0/1057536992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae01012f0 0x7f8ae0107ba0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800b0d0 tx=0x7f8ad800b3e0 comp rx=0 tx=0).stop 2023-07-20T17:48:20.044 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.038+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/1057536992 shutdown_connections 2023-07-20T17:48:20.044 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.038+0000 7f8ae4d1d700 1 --2- 172.21.15.49:0/1057536992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ae00ff760 0x7f8ae00ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:20.044 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.038+0000 7f8ae4d1d700 1 --2- 172.21.15.49:0/1057536992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae01012f0 0x7f8ae0107ba0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:20.045 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.038+0000 7f8ae4d1d700 1 --2- 172.21.15.49:0/1057536992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ae0100950 0x7f8ae0100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:20.045 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.038+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/1057536992 >> 172.21.15.49:0/1057536992 conn(0x7f8ae00facd0 msgr2=0x7f8ae00fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:20.045 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.038+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/1057536992 shutdown_connections 2023-07-20T17:48:20.045 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.038+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/1057536992 wait complete. 2023-07-20T17:48:20.045 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8ae4d1d700 1 Processor -- start 2023-07-20T17:48:20.045 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8ae4d1d700 1 -- start start 2023-07-20T17:48:20.046 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8ae4d1d700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ae00ff760 0x7f8ae0195580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:20.046 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8ae4d1d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ae0100950 0x7f8ae0195ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:20.046 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8ae4d1d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae01012f0 0x7f8ae019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:20.046 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8ae4d1d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae019d100 con 0x7f8ae01012f0 2023-07-20T17:48:20.046 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8ae4d1d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae019d240 con 0x7f8ae0100950 2023-07-20T17:48:20.047 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8ae4d1d700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae019d3d0 con 0x7f8ae00ff760 2023-07-20T17:48:20.047 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8aded9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae01012f0 0x7f8ae019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:20.047 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8aded9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae01012f0 0x7f8ae019cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36168/0 (socket says 172.21.15.49:36168) 2023-07-20T17:48:20.047 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8aded9d700 1 -- 172.21.15.49:0/2923461690 learned_addr learned my addr 172.21.15.49:0/2923461690 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:20.047 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8ade59c700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ae00ff760 0x7f8ae0195580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:20.047 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.039+0000 7f8addd9b700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ae0100950 0x7f8ae0195ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:20.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.040+0000 7f8aded9d700 1 -- 172.21.15.49:0/2923461690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ae00ff760 msgr2=0x7f8ae0195580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:20.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.040+0000 7f8aded9d700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ae00ff760 0x7f8ae0195580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:20.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.040+0000 7f8aded9d700 1 -- 172.21.15.49:0/2923461690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ae0100950 msgr2=0x7f8ae0195ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:20.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.040+0000 7f8aded9d700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ae0100950 0x7f8ae0195ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:20.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.040+0000 7f8aded9d700 1 -- 172.21.15.49:0/2923461690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad8009d00 con 0x7f8ae01012f0 2023-07-20T17:48:20.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.040+0000 7f8aded9d700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae01012f0 0x7f8ae019cb30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8008eb0 tx=0x7f8ad8008ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:20.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.040+0000 7f8ad77fe700 1 -- 172.21.15.49:0/2923461690 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad800e040 con 0x7f8ae01012f0 2023-07-20T17:48:20.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.040+0000 7f8ad77fe700 1 -- 172.21.15.49:0/2923461690 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ad800bdb0 con 0x7f8ae01012f0 2023-07-20T17:48:20.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.040+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/2923461690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae019d650 con 0x7f8ae01012f0 2023-07-20T17:48:20.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.040+0000 7f8ad77fe700 1 -- 172.21.15.49:0/2923461690 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad801d470 con 0x7f8ae01012f0 2023-07-20T17:48:20.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.040+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/2923461690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae019db20 con 0x7f8ae01012f0 2023-07-20T17:48:20.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.041+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/2923461690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae0106cb0 con 0x7f8ae01012f0 2023-07-20T17:48:20.050 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.042+0000 7f8ad77fe700 1 -- 172.21.15.49:0/2923461690 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8ad8005020 con 0x7f8ae01012f0 2023-07-20T17:48:20.051 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.042+0000 7f8ad77fe700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f8acc0a11d0 0x7f8acc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:20.051 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.043+0000 7f8ade59c700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f8acc0a11d0 0x7f8acc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:20.051 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.043+0000 7f8ad77fe700 1 -- 172.21.15.49:0/2923461690 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8ad80cd420 con 0x7f8ae01012f0 2023-07-20T17:48:20.051 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.043+0000 7f8ade59c700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f8acc0a11d0 0x7f8acc0a3690 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8ae01007b0 tx=0x7f8ad0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:20.051 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.047+0000 7f8ad77fe700 1 -- 172.21.15.49:0/2923461690 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8ad8097000 con 0x7f8ae01012f0 2023-07-20T17:48:20.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.291+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/2923461690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ae002c970 con 0x7f8ae01012f0 2023-07-20T17:48:20.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.291+0000 7f8ad77fe700 1 -- 172.21.15.49:0/2923461690 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f8ad809a620 con 0x7f8ae01012f0 2023-07-20T17:48:20.293 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:20.294 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-20T17:48:20.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.293+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/2923461690 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f8acc0a11d0 msgr2=0x7f8acc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:20.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.293+0000 7f8ae4d1d700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f8acc0a11d0 0x7f8acc0a3690 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8ae01007b0 tx=0x7f8ad0009260 comp rx=0 tx=0).stop 2023-07-20T17:48:20.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.294+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/2923461690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae01012f0 msgr2=0x7f8ae019cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:20.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.294+0000 7f8ae4d1d700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae01012f0 0x7f8ae019cb30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8008eb0 tx=0x7f8ad8008ee0 comp rx=0 tx=0).stop 2023-07-20T17:48:20.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.294+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/2923461690 shutdown_connections 2023-07-20T17:48:20.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.294+0000 7f8ae4d1d700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f8acc0a11d0 0x7f8acc0a3690 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:20.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.294+0000 7f8ae4d1d700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ae00ff760 0x7f8ae0195580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:20.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.294+0000 7f8ae4d1d700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae01012f0 0x7f8ae019cb30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:20.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.294+0000 7f8ae4d1d700 1 --2- 172.21.15.49:0/2923461690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ae0100950 0x7f8ae0195ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:20.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.294+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/2923461690 >> 172.21.15.49:0/2923461690 conn(0x7f8ae00facd0 msgr2=0x7f8ae0103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:20.297 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.294+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/2923461690 shutdown_connections 2023-07-20T17:48:20.297 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:20.294+0000 7f8ae4d1d700 1 -- 172.21.15.49:0/2923461690 wait complete. 2023-07-20T17:48:20.297 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 1 2023-07-20T17:48:20.369 INFO:journalctl@ceph.mds.l.smithi097.stdout:Jul 20 17:48:20 smithi097 systemd[1]: Starting Ceph mds.l for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:20.627 INFO:journalctl@ceph.mds.l.smithi097.stdout:Jul 20 17:48:20 smithi097 podman[123509]: 2023-07-20T17:48:20.628 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:20 smithi097 ceph-mon[98398]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:20.628 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:20 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/529204552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:20.628 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:20 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1980352675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T17:48:20.629 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:20 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/2923461690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:20.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:20 smithi049 ceph-mon[96530]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:20.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:20 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/529204552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:20.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:20 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1980352675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T17:48:20.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:20 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2923461690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:20.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:20 smithi079 ceph-mon[98036]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:20.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:20 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/529204552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:20.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:20 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1980352675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T17:48:20.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:20 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/2923461690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:21.018 INFO:journalctl@ceph.mds.l.smithi097.stdout:Jul 20 17:48:20 smithi097 ceph-mds[123549]: -- 172.21.15.97:0/3439611221 <== mon.1 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5652afffd1e0 con 0x5652affd2c00 2023-07-20T17:48:21.018 INFO:journalctl@ceph.mds.l.smithi097.stdout:Jul 20 17:48:20 smithi097 bash[123509]: dfbc58e7ee6dc6fe927285a06774f2bce8f9f758f06a5b14ee43549d7181a47e 2023-07-20T17:48:21.018 INFO:journalctl@ceph.mds.l.smithi097.stdout:Jul 20 17:48:20 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-l[123525]: starting mds.l at 2023-07-20T17:48:21.018 INFO:journalctl@ceph.mds.l.smithi097.stdout:Jul 20 17:48:20 smithi097 systemd[1]: Started Ceph mds.l for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:21.710 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:21 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.97:6832/1293946253,v1:172.21.15.97:6833/1293946253] up:boot 2023-07-20T17:48:21.710 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:21 smithi079 ceph-mon[98036]: fsmap 1 up:standby 2023-07-20T17:48:21.711 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T17:48:21.711 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:21.711 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:21.711 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:21.711 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:21.711 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:21.711 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:21 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:21.711 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:21 smithi079 ceph-mon[98036]: Deploying daemon mds.k on smithi079 2023-07-20T17:48:21.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:21 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.97:6832/1293946253,v1:172.21.15.97:6833/1293946253] up:boot 2023-07-20T17:48:21.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:21 smithi049 ceph-mon[96530]: fsmap 1 up:standby 2023-07-20T17:48:21.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T17:48:21.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:21.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:21.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:21.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:21.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:21.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:21 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:21.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:21 smithi049 ceph-mon[96530]: Deploying daemon mds.k on smithi079 2023-07-20T17:48:22.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:21 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.97:6832/1293946253,v1:172.21.15.97:6833/1293946253] up:boot 2023-07-20T17:48:22.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:21 smithi097 ceph-mon[98398]: fsmap 1 up:standby 2023-07-20T17:48:22.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T17:48:22.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:22.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:22.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:22.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:22.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:22.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:21 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:22.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:21 smithi097 ceph-mon[98398]: Deploying daemon mds.k on smithi079 2023-07-20T17:48:22.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:22.317 INFO:journalctl@ceph.mds.k.smithi079.stdout:Jul 20 17:48:22 smithi079 systemd[1]: Starting Ceph mds.k for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:22.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.447+0000 7fe375528700 1 -- 172.21.15.49:0/3618024735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3701019f0 msgr2=0x7fe370101e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:22.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.447+0000 7fe375528700 1 --2- 172.21.15.49:0/3618024735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3701019f0 0x7fe370101e50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe3580099c0 tx=0x7fe358009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:22.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.447+0000 7fe375528700 1 -- 172.21.15.49:0/3618024735 shutdown_connections 2023-07-20T17:48:22.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.447+0000 7fe375528700 1 --2- 172.21.15.49:0/3618024735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe370100800 0x7fe370100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:22.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.447+0000 7fe375528700 1 --2- 172.21.15.49:0/3618024735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe370102390 0x7fe370108c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:22.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.447+0000 7fe375528700 1 --2- 172.21.15.49:0/3618024735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3701019f0 0x7fe370101e50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:22.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.447+0000 7fe375528700 1 -- 172.21.15.49:0/3618024735 >> 172.21.15.49:0/3618024735 conn(0x7fe3700fbd30 msgr2=0x7fe3700fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:22.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.447+0000 7fe375528700 1 -- 172.21.15.49:0/3618024735 shutdown_connections 2023-07-20T17:48:22.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.447+0000 7fe375528700 1 -- 172.21.15.49:0/3618024735 wait complete. 2023-07-20T17:48:22.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe375528700 1 Processor -- start 2023-07-20T17:48:22.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe375528700 1 -- start start 2023-07-20T17:48:22.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe375528700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe370100800 0x7fe370196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:22.452 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe375528700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3701019f0 0x7fe370196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:22.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe375528700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe370102390 0x7fe37019dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:22.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe375528700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37019e0b0 con 0x7fe3701019f0 2023-07-20T17:48:22.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe375528700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37019e1f0 con 0x7fe370102390 2023-07-20T17:48:22.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe375528700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37019e380 con 0x7fe370100800 2023-07-20T17:48:22.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe36e7fc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3701019f0 0x7fe370196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:22.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe36e7fc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3701019f0 0x7fe370196a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36180/0 (socket says 172.21.15.49:36180) 2023-07-20T17:48:22.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe36e7fc700 1 -- 172.21.15.49:0/665800370 learned_addr learned my addr 172.21.15.49:0/665800370 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:22.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.448+0000 7fe36effd700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe370100800 0x7fe370196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:22.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe36f7fe700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe370102390 0x7fe37019dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:22.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe36e7fc700 1 -- 172.21.15.49:0/665800370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe370100800 msgr2=0x7fe370196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:22.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe36e7fc700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe370100800 0x7fe370196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:22.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe36e7fc700 1 -- 172.21.15.49:0/665800370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe370102390 msgr2=0x7fe37019dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:22.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe36e7fc700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe370102390 0x7fe37019dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:22.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe36e7fc700 1 -- 172.21.15.49:0/665800370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3580095f0 con 0x7fe3701019f0 2023-07-20T17:48:22.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe36e7fc700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3701019f0 0x7fe370196a70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe358009990 tx=0x7fe358004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:22.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe367fff700 1 -- 172.21.15.49:0/665800370 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35801d070 con 0x7fe3701019f0 2023-07-20T17:48:22.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe367fff700 1 -- 172.21.15.49:0/665800370 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe358004090 con 0x7fe3701019f0 2023-07-20T17:48:22.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe375528700 1 -- 172.21.15.49:0/665800370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37019e600 con 0x7fe3701019f0 2023-07-20T17:48:22.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe367fff700 1 -- 172.21.15.49:0/665800370 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358004a00 con 0x7fe3701019f0 2023-07-20T17:48:22.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.449+0000 7fe375528700 1 -- 172.21.15.49:0/665800370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe37019eab0 con 0x7fe3701019f0 2023-07-20T17:48:22.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.450+0000 7fe375528700 1 -- 172.21.15.49:0/665800370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe370107e60 con 0x7fe3701019f0 2023-07-20T17:48:22.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.451+0000 7fe367fff700 1 -- 172.21.15.49:0/665800370 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe358003bd0 con 0x7fe3701019f0 2023-07-20T17:48:22.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.452+0000 7fe367fff700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fe35c0a1150 0x7fe35c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:22.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.452+0000 7fe367fff700 1 -- 172.21.15.49:0/665800370 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe3580cd790 con 0x7fe3701019f0 2023-07-20T17:48:22.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.452+0000 7fe36effd700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fe35c0a1150 0x7fe35c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:22.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.452+0000 7fe36effd700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fe35c0a1150 0x7fe35c0a3610 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe370101850 tx=0x7fe36000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:22.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.455+0000 7fe367fff700 1 -- 172.21.15.49:0/665800370 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe358097410 con 0x7fe3701019f0 2023-07-20T17:48:22.705 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.703+0000 7fe375528700 1 -- 172.21.15.49:0/665800370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe37002c970 con 0x7fe3701019f0 2023-07-20T17:48:22.706 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.704+0000 7fe367fff700 1 -- 172.21.15.49:0/665800370 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1257 (secure 0 0 0) 0x7fe35809aa30 con 0x7fe3701019f0 2023-07-20T17:48:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:22.706 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/1293946253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1293946253},{"type":"v1","addr":"172.21.15.97:6833","nonce":1293946253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T17:48:22.708 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 -- 172.21.15.49:0/665800370 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fe35c0a1150 msgr2=0x7fe35c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:22.708 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fe35c0a1150 0x7fe35c0a3610 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe370101850 tx=0x7fe36000b410 comp rx=0 tx=0).stop 2023-07-20T17:48:22.708 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 -- 172.21.15.49:0/665800370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3701019f0 msgr2=0x7fe370196a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:22.708 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3701019f0 0x7fe370196a70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe358009990 tx=0x7fe358004730 comp rx=0 tx=0).stop 2023-07-20T17:48:22.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 -- 172.21.15.49:0/665800370 shutdown_connections 2023-07-20T17:48:22.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fe35c0a1150 0x7fe35c0a3610 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:22.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe370100800 0x7fe370196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:22.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe370102390 0x7fe37019dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:22.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 --2- 172.21.15.49:0/665800370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3701019f0 0x7fe370196a70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:22.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 -- 172.21.15.49:0/665800370 >> 172.21.15.49:0/665800370 conn(0x7fe3700fbd30 msgr2=0x7fe370104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:22.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 -- 172.21.15.49:0/665800370 shutdown_connections 2023-07-20T17:48:22.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:22.706+0000 7fe375528700 1 -- 172.21.15.49:0/665800370 wait complete. 2023-07-20T17:48:22.710 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 2 2023-07-20T17:48:22.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:22 smithi079 ceph-mon[98036]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:22.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:22 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/665800370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:22.818 INFO:journalctl@ceph.mds.k.smithi079.stdout:Jul 20 17:48:22 smithi079 podman[122298]: 2023-07-20T17:48:22.819 INFO:journalctl@ceph.mds.k.smithi079.stdout:Jul 20 17:48:22 smithi079 bash[122298]: 3b5ebf871f03dc307d790b8e569d9c89ae926b66890f75bf16aa225157e41113 2023-07-20T17:48:22.819 INFO:journalctl@ceph.mds.k.smithi079.stdout:Jul 20 17:48:22 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-k[122314]: starting mds.k at 2023-07-20T17:48:22.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:22 smithi049 ceph-mon[96530]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:22.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:22 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/665800370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:23.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:22 smithi097 ceph-mon[98398]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:23.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:22 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/665800370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:23.317 INFO:journalctl@ceph.mds.k.smithi079.stdout:Jul 20 17:48:22 smithi079 systemd[1]: Started Ceph mds.k for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:24.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:23 smithi049 ceph-mon[96530]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:24.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:24.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:24.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:24.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:24.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:24.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:24.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:23 smithi049 ceph-mon[96530]: Deploying daemon mds.j on smithi049 2023-07-20T17:48:24.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:23 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.79:6832/2111036634,v1:172.21.15.79:6833/2111036634] up:boot 2023-07-20T17:48:24.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:23 smithi049 ceph-mon[96530]: fsmap 2 up:standby 2023-07-20T17:48:24.074 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:23 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T17:48:24.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:23 smithi097 ceph-mon[98398]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:24.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:24.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:24.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:24.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:24.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:24.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:24.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:23 smithi097 ceph-mon[98398]: Deploying daemon mds.j on smithi049 2023-07-20T17:48:24.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:23 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.79:6832/2111036634,v1:172.21.15.79:6833/2111036634] up:boot 2023-07-20T17:48:24.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:23 smithi097 ceph-mon[98398]: fsmap 2 up:standby 2023-07-20T17:48:24.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:23 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T17:48:24.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:23 smithi079 ceph-mon[98036]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:24.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:24.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:24.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:24.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:24.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:24.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:24.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:23 smithi079 ceph-mon[98036]: Deploying daemon mds.j on smithi049 2023-07-20T17:48:24.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:23 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.79:6832/2111036634,v1:172.21.15.79:6833/2111036634] up:boot 2023-07-20T17:48:24.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:23 smithi079 ceph-mon[98036]: fsmap 2 up:standby 2023-07-20T17:48:24.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:23 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T17:48:24.644 INFO:journalctl@ceph.mds.j.smithi049.stdout:Jul 20 17:48:24 smithi049 systemd[1]: Starting Ceph mds.j for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:24.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:24.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.853+0000 7fa2815d3700 1 -- 172.21.15.49:0/1110205519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c106600 msgr2=0x7fa27c1089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:24.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.853+0000 7fa2815d3700 1 --2- 172.21.15.49:0/1110205519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c106600 0x7fa27c1089f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa27400b0a0 tx=0x7fa27400b3b0 comp rx=0 tx=0).stop 2023-07-20T17:48:24.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.853+0000 7fa2815d3700 1 -- 172.21.15.49:0/1110205519 shutdown_connections 2023-07-20T17:48:24.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.853+0000 7fa2815d3700 1 --2- 172.21.15.49:0/1110205519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c106600 0x7fa27c1089f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:24.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.853+0000 7fa2815d3700 1 --2- 172.21.15.49:0/1110205519 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa27c1013a0 0x7fa27c103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:24.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.853+0000 7fa2815d3700 1 --2- 172.21.15.49:0/1110205519 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa27c103cd0 0x7fa27c1060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:24.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.853+0000 7fa2815d3700 1 -- 172.21.15.49:0/1110205519 >> 172.21.15.49:0/1110205519 conn(0x7fa27c0facd0 msgr2=0x7fa27c0fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:24.857 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.853+0000 7fa2815d3700 1 -- 172.21.15.49:0/1110205519 shutdown_connections 2023-07-20T17:48:24.857 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.853+0000 7fa2815d3700 1 -- 172.21.15.49:0/1110205519 wait complete. 2023-07-20T17:48:24.858 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.854+0000 7fa2815d3700 1 Processor -- start 2023-07-20T17:48:24.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.854+0000 7fa2815d3700 1 -- start start 2023-07-20T17:48:24.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa2815d3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa27c1013a0 0x7fa27c100d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:24.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa2815d3700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa27c103cd0 0x7fa27c0ff350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:24.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa2815d3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c106600 0x7fa27c0ff890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:24.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa2815d3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c100030 con 0x7fa27c106600 2023-07-20T17:48:24.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa2815d3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c1001a0 con 0x7fa27c1013a0 2023-07-20T17:48:24.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa2815d3700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c1a0640 con 0x7fa27c103cd0 2023-07-20T17:48:24.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27b7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c106600 0x7fa27c0ff890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:24.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27b7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c106600 0x7fa27c0ff890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36194/0 (socket says 172.21.15.49:36194) 2023-07-20T17:48:24.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27b7fe700 1 -- 172.21.15.49:0/2953838932 learned_addr learned my addr 172.21.15.49:0/2953838932 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:24.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27affd700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa27c1013a0 0x7fa27c100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:24.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27a7fc700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa27c103cd0 0x7fa27c0ff350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:24.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27b7fe700 1 -- 172.21.15.49:0/2953838932 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa27c103cd0 msgr2=0x7fa27c0ff350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:24.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27b7fe700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa27c103cd0 0x7fa27c0ff350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:24.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27b7fe700 1 -- 172.21.15.49:0/2953838932 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa27c1013a0 msgr2=0x7fa27c100d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:24.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27b7fe700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa27c1013a0 0x7fa27c100d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:24.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27b7fe700 1 -- 172.21.15.49:0/2953838932 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa274009d00 con 0x7fa27c106600 2023-07-20T17:48:24.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27affd700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa27c1013a0 0x7fa27c100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:48:24.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa27b7fe700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c106600 0x7fa27c0ff890 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa274009fd0 tx=0x7fa27400bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:24.862 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa263fff700 1 -- 172.21.15.49:0/2953838932 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27400e040 con 0x7fa27c106600 2023-07-20T17:48:24.862 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa263fff700 1 -- 172.21.15.49:0/2953838932 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa274003e10 con 0x7fa27c106600 2023-07-20T17:48:24.862 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.855+0000 7fa2815d3700 1 -- 172.21.15.49:0/2953838932 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27c1a08c0 con 0x7fa27c106600 2023-07-20T17:48:24.862 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.856+0000 7fa263fff700 1 -- 172.21.15.49:0/2953838932 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27401b7e0 con 0x7fa27c106600 2023-07-20T17:48:24.862 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.856+0000 7fa2815d3700 1 -- 172.21.15.49:0/2953838932 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27c1a0ce0 con 0x7fa27c106600 2023-07-20T17:48:24.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.857+0000 7fa2815d3700 1 -- 172.21.15.49:0/2953838932 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27c0fc850 con 0x7fa27c106600 2023-07-20T17:48:24.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.858+0000 7fa263fff700 1 -- 172.21.15.49:0/2953838932 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa27401da30 con 0x7fa27c106600 2023-07-20T17:48:24.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.859+0000 7fa263fff700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa2640a1150 0x7fa2640a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:24.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.859+0000 7fa27affd700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa2640a1150 0x7fa2640a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:24.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.859+0000 7fa263fff700 1 -- 172.21.15.49:0/2953838932 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa2740cd720 con 0x7fa27c106600 2023-07-20T17:48:24.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.859+0000 7fa27affd700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa2640a1150 0x7fa2640a3610 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa26c005fd0 tx=0x7fa26c005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:24.865 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:24.863+0000 7fa263fff700 1 -- 172.21.15.49:0/2953838932 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa2740973a0 con 0x7fa27c106600 2023-07-20T17:48:24.990 INFO:journalctl@ceph.mds.j.smithi049.stdout:Jul 20 17:48:24 smithi049 podman[129983]: 2023-07-20T17:48:24.990 INFO:journalctl@ceph.mds.j.smithi049.stdout:Jul 20 17:48:24 smithi049 bash[129983]: 90adec27bb493b675c4c5417d25cd52a969849989c19c8596296794fc3a06644 2023-07-20T17:48:24.990 INFO:journalctl@ceph.mds.j.smithi049.stdout:Jul 20 17:48:24 smithi049 ceph-mds[130047]: -- 172.21.15.49:0/2501557053 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f2bb6531e0 con 0x55f2bb628800 2023-07-20T17:48:24.991 INFO:journalctl@ceph.mds.j.smithi049.stdout:Jul 20 17:48:24 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-j[129999]: starting mds.j at 2023-07-20T17:48:24.991 INFO:journalctl@ceph.mds.j.smithi049.stdout:Jul 20 17:48:24 smithi049 systemd[1]: Started Ceph mds.j for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:25.121 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.118+0000 7fa2815d3700 1 -- 172.21.15.49:0/2953838932 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa27c02c970 con 0x7fa27c106600 2023-07-20T17:48:25.121 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.119+0000 7fa263fff700 1 -- 172.21.15.49:0/2953838932 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2014 (secure 0 0 0) 0x7fa27409a9c0 con 0x7fa27c106600 2023-07-20T17:48:25.122 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:25.122 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/1293946253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1293946253},{"type":"v1","addr":"172.21.15.97:6833","nonce":1293946253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2111036634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2111036634},{"type":"v1","addr":"172.21.15.79:6833","nonce":2111036634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-20T17:48:25.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.122+0000 7fa2815d3700 1 -- 172.21.15.49:0/2953838932 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa2640a1150 msgr2=0x7fa2640a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:25.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.122+0000 7fa2815d3700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa2640a1150 0x7fa2640a3610 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa26c005fd0 tx=0x7fa26c005f60 comp rx=0 tx=0).stop 2023-07-20T17:48:25.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.122+0000 7fa2815d3700 1 -- 172.21.15.49:0/2953838932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c106600 msgr2=0x7fa27c0ff890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:25.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.122+0000 7fa2815d3700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c106600 0x7fa27c0ff890 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa274009fd0 tx=0x7fa27400bd90 comp rx=0 tx=0).stop 2023-07-20T17:48:25.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.122+0000 7fa2815d3700 1 -- 172.21.15.49:0/2953838932 shutdown_connections 2023-07-20T17:48:25.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.122+0000 7fa2815d3700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa2640a1150 0x7fa2640a3610 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:25.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.122+0000 7fa2815d3700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c106600 0x7fa27c0ff890 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:25.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.122+0000 7fa2815d3700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa27c1013a0 0x7fa27c100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:25.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.122+0000 7fa2815d3700 1 --2- 172.21.15.49:0/2953838932 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa27c103cd0 0x7fa27c0ff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:25.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.122+0000 7fa2815d3700 1 -- 172.21.15.49:0/2953838932 >> 172.21.15.49:0/2953838932 conn(0x7fa27c0facd0 msgr2=0x7fa27c102010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:25.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.122+0000 7fa2815d3700 1 -- 172.21.15.49:0/2953838932 shutdown_connections 2023-07-20T17:48:25.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:25.123+0000 7fa2815d3700 1 -- 172.21.15.49:0/2953838932 wait complete. 2023-07-20T17:48:25.127 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 3 2023-07-20T17:48:26.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:26.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:26.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:26.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:26.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:26.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:26.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:26.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: Deploying daemon mds.d on smithi049 2023-07-20T17:48:26.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2953838932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:26.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.49:6834/1572141770,v1:172.21.15.49:6835/1572141770] up:boot 2023-07-20T17:48:26.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: fsmap 3 up:standby 2023-07-20T17:48:26.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:25 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T17:48:26.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:26.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:26.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:26.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:26.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:26.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:26.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:26.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: Deploying daemon mds.d on smithi049 2023-07-20T17:48:26.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/2953838932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:26.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.49:6834/1572141770,v1:172.21.15.49:6835/1572141770] up:boot 2023-07-20T17:48:26.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: fsmap 3 up:standby 2023-07-20T17:48:26.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:25 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T17:48:26.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:26.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:26.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:26.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:26.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:26.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:26.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:26.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: Deploying daemon mds.d on smithi049 2023-07-20T17:48:26.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/2953838932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:26.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.49:6834/1572141770,v1:172.21.15.49:6835/1572141770] up:boot 2023-07-20T17:48:26.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: fsmap 3 up:standby 2023-07-20T17:48:26.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:25 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T17:48:26.741 INFO:journalctl@ceph.mds.d.smithi049.stdout:Jul 20 17:48:26 smithi049 systemd[1]: Starting Ceph mds.d for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:27.041 INFO:journalctl@ceph.mds.d.smithi049.stdout:Jul 20 17:48:26 smithi049 podman[130369]: 2023-07-20T17:48:27.041 INFO:journalctl@ceph.mds.d.smithi049.stdout:Jul 20 17:48:26 smithi049 ceph-mds[130410]: -- 172.21.15.49:0/2487101458 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5647a0ce51e0 con 0x5647a0cbac00 2023-07-20T17:48:27.041 INFO:journalctl@ceph.mds.d.smithi049.stdout:Jul 20 17:48:26 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-d[130385]: starting mds.d at 2023-07-20T17:48:27.041 INFO:journalctl@ceph.mds.d.smithi049.stdout:Jul 20 17:48:26 smithi049 bash[130369]: 894aa98730ecf8ee95df428ade40495e418dcebdbde77231b4278b458bd09970 2023-07-20T17:48:27.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:27.276 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.273+0000 7f624dc1f700 1 -- 172.21.15.49:0/908913650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6248100950 msgr2=0x7f6248100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:27.276 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.273+0000 7f624dc1f700 1 --2- 172.21.15.49:0/908913650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6248100950 0x7f6248100db0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f62380099c0 tx=0x7f6238009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:27.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.273+0000 7f624dc1f700 1 -- 172.21.15.49:0/908913650 shutdown_connections 2023-07-20T17:48:27.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.273+0000 7f624dc1f700 1 --2- 172.21.15.49:0/908913650 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62481012f0 0x7f6248107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:27.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.273+0000 7f624dc1f700 1 --2- 172.21.15.49:0/908913650 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f62480ff760 0x7f62480ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:27.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.273+0000 7f624dc1f700 1 --2- 172.21.15.49:0/908913650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6248100950 0x7f6248100db0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:27.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.273+0000 7f624dc1f700 1 -- 172.21.15.49:0/908913650 >> 172.21.15.49:0/908913650 conn(0x7f62480facd0 msgr2=0x7f62480fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:27.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.273+0000 7f624dc1f700 1 -- 172.21.15.49:0/908913650 shutdown_connections 2023-07-20T17:48:27.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.273+0000 7f624dc1f700 1 -- 172.21.15.49:0/908913650 wait complete. 2023-07-20T17:48:27.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f624dc1f700 1 Processor -- start 2023-07-20T17:48:27.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f624dc1f700 1 -- start start 2023-07-20T17:48:27.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f624dc1f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62480ff760 0x7f6248196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:27.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f624dc1f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6248100950 0x7f6248196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:27.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f624dc1f700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62481012f0 0x7f624819dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:27.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f624dc1f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624819e0a0 con 0x7f62480ff760 2023-07-20T17:48:27.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f624dc1f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624819e1e0 con 0x7f6248100950 2023-07-20T17:48:27.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f624dc1f700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624819e370 con 0x7f62481012f0 2023-07-20T17:48:27.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f62477fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62480ff760 0x7f6248196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:27.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f62477fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62480ff760 0x7f6248196520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36230/0 (socket says 172.21.15.49:36230) 2023-07-20T17:48:27.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f62477fe700 1 -- 172.21.15.49:0/3842520735 learned_addr learned my addr 172.21.15.49:0/3842520735 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:27.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f6246ffd700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6248100950 0x7f6248196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:27.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f6247fff700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62481012f0 0x7f624819dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:27.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f62477fe700 1 -- 172.21.15.49:0/3842520735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62481012f0 msgr2=0x7f624819dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:27.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f62477fe700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62481012f0 0x7f624819dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:27.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f62477fe700 1 -- 172.21.15.49:0/3842520735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6248100950 msgr2=0x7f6248196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:27.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f62477fe700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6248100950 0x7f6248196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:27.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.274+0000 7f62477fe700 1 -- 172.21.15.49:0/3842520735 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62380095f0 con 0x7f62480ff760 2023-07-20T17:48:27.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.275+0000 7f62477fe700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62480ff760 0x7f6248196520 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f623c00d980 tx=0x7f623c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:27.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.275+0000 7f6244ff9700 1 -- 172.21.15.49:0/3842520735 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c004020 con 0x7f62480ff760 2023-07-20T17:48:27.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.275+0000 7f624dc1f700 1 -- 172.21.15.49:0/3842520735 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624819e650 con 0x7f62480ff760 2023-07-20T17:48:27.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.275+0000 7f6244ff9700 1 -- 172.21.15.49:0/3842520735 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f623c00f690 con 0x7f62480ff760 2023-07-20T17:48:27.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.275+0000 7f6244ff9700 1 -- 172.21.15.49:0/3842520735 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c003a60 con 0x7f62480ff760 2023-07-20T17:48:27.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.275+0000 7f624dc1f700 1 -- 172.21.15.49:0/3842520735 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f624819eb60 con 0x7f62480ff760 2023-07-20T17:48:27.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.276+0000 7f624dc1f700 1 -- 172.21.15.49:0/3842520735 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f624810afd0 con 0x7f62480ff760 2023-07-20T17:48:27.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.277+0000 7f6244ff9700 1 -- 172.21.15.49:0/3842520735 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f623c00fb00 con 0x7f62480ff760 2023-07-20T17:48:27.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.277+0000 7f6244ff9700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f62300a11d0 0x7f62300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:27.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.277+0000 7f6246ffd700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f62300a11d0 0x7f62300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:27.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.277+0000 7f6244ff9700 1 -- 172.21.15.49:0/3842520735 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f623c0cc440 con 0x7f62480ff760 2023-07-20T17:48:27.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.278+0000 7f6246ffd700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f62300a11d0 0x7f62300a3690 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6238000c00 tx=0x7f6238009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:27.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.281+0000 7f6244ff9700 1 -- 172.21.15.49:0/3842520735 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f623c095f90 con 0x7f62480ff760 2023-07-20T17:48:27.490 INFO:journalctl@ceph.mds.d.smithi049.stdout:Jul 20 17:48:27 smithi049 systemd[1]: Started Ceph mds.d for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:27.526 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.524+0000 7f624dc1f700 1 -- 172.21.15.49:0/3842520735 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f624819f200 con 0x7f62480ff760 2023-07-20T17:48:27.527 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.524+0000 7f6244ff9700 1 -- 172.21.15.49:0/3842520735 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2771 (secure 0 0 0) 0x7f623c0995b0 con 0x7f62480ff760 2023-07-20T17:48:27.527 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:27.527 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1572141770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1572141770},{"type":"v1","addr":"172.21.15.49:6835","nonce":1572141770}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/1293946253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1293946253},{"type":"v1","addr":"172.21.15.97:6833","nonce":1293946253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2111036634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2111036634},{"type":"v1","addr":"172.21.15.79:6833","nonce":2111036634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-20T17:48:27.529 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 -- 172.21.15.49:0/3842520735 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f62300a11d0 msgr2=0x7f62300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:27.529 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f62300a11d0 0x7f62300a3690 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6238000c00 tx=0x7f6238009f10 comp rx=0 tx=0).stop 2023-07-20T17:48:27.529 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 -- 172.21.15.49:0/3842520735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62480ff760 msgr2=0x7f6248196520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:27.529 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62480ff760 0x7f6248196520 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f623c00d980 tx=0x7f623c00dc90 comp rx=0 tx=0).stop 2023-07-20T17:48:27.529 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 -- 172.21.15.49:0/3842520735 shutdown_connections 2023-07-20T17:48:27.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f62300a11d0 0x7f62300a3690 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:27.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62481012f0 0x7f624819dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:27.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62480ff760 0x7f6248196520 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:27.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 --2- 172.21.15.49:0/3842520735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6248100950 0x7f6248196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:27.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 -- 172.21.15.49:0/3842520735 >> 172.21.15.49:0/3842520735 conn(0x7f62480facd0 msgr2=0x7f6248068780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:27.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 -- 172.21.15.49:0/3842520735 shutdown_connections 2023-07-20T17:48:27.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:27.526+0000 7f624dc1f700 1 -- 172.21.15.49:0/3842520735 wait complete. 2023-07-20T17:48:27.531 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 4 2023-07-20T17:48:28.293 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:28.294 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:28.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:28.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:28.295 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:28.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:28.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:28.296 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: Deploying daemon mds.e on smithi079 2023-07-20T17:48:28.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3842520735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:28.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.49:6836/4279110827,v1:172.21.15.49:6837/4279110827] up:boot 2023-07-20T17:48:28.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: fsmap 4 up:standby 2023-07-20T17:48:28.297 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:28 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T17:48:28.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:28.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:28.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:28.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:28.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:28.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:28.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:28.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: Deploying daemon mds.e on smithi079 2023-07-20T17:48:28.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3842520735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:28.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.49:6836/4279110827,v1:172.21.15.49:6837/4279110827] up:boot 2023-07-20T17:48:28.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: fsmap 4 up:standby 2023-07-20T17:48:28.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:28 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T17:48:28.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:28.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:28.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:28.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:28.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:28.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:28.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:28.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: Deploying daemon mds.e on smithi079 2023-07-20T17:48:28.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3842520735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:28.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.49:6836/4279110827,v1:172.21.15.49:6837/4279110827] up:boot 2023-07-20T17:48:28.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: fsmap 4 up:standby 2023-07-20T17:48:28.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:28 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T17:48:28.932 INFO:journalctl@ceph.mds.e.smithi079.stdout:Jul 20 17:48:28 smithi079 systemd[1]: Starting Ceph mds.e for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:29.228 INFO:journalctl@ceph.mds.e.smithi079.stdout:Jul 20 17:48:28 smithi079 podman[122637]: 2023-07-20T17:48:29.230 INFO:journalctl@ceph.mds.e.smithi079.stdout:Jul 20 17:48:29 smithi079 bash[122637]: 0abb7003a6e2a4ba8057d766143becc94f36c09da3f794bbbfa7f57dd13d4f0b 2023-07-20T17:48:29.231 INFO:journalctl@ceph.mds.e.smithi079.stdout:Jul 20 17:48:29 smithi079 ceph-mds[122677]: -- 172.21.15.79:0/3514603022 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56276c3e11e0 con 0x56276c3b6800 2023-07-20T17:48:29.231 INFO:journalctl@ceph.mds.e.smithi079.stdout:Jul 20 17:48:29 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-e[122653]: starting mds.e at 2023-07-20T17:48:29.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:29.567 INFO:journalctl@ceph.mds.e.smithi079.stdout:Jul 20 17:48:29 smithi079 systemd[1]: Started Ceph mds.e for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:29.683 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.680+0000 7f3b50af0700 1 -- 172.21.15.49:0/1825819208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b4c0ff720 msgr2=0x7f3b4c0ffb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:29.683 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.680+0000 7f3b50af0700 1 --2- 172.21.15.49:0/1825819208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b4c0ff720 0x7f3b4c0ffb40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0099c0 tx=0x7f3b3c009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.680+0000 7f3b50af0700 1 -- 172.21.15.49:0/1825819208 shutdown_connections 2023-07-20T17:48:29.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.680+0000 7f3b50af0700 1 --2- 172.21.15.49:0/1825819208 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c1012b0 0x7f3b4c107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.680+0000 7f3b50af0700 1 --2- 172.21.15.49:0/1825819208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b4c0ff720 0x7f3b4c0ffb40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.680+0000 7f3b50af0700 1 --2- 172.21.15.49:0/1825819208 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4c100910 0x7f3b4c100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.680+0000 7f3b50af0700 1 -- 172.21.15.49:0/1825819208 >> 172.21.15.49:0/1825819208 conn(0x7f3b4c0facb0 msgr2=0x7f3b4c0fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:29.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.680+0000 7f3b50af0700 1 -- 172.21.15.49:0/1825819208 shutdown_connections 2023-07-20T17:48:29.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.680+0000 7f3b50af0700 1 -- 172.21.15.49:0/1825819208 wait complete. 2023-07-20T17:48:29.686 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b50af0700 1 Processor -- start 2023-07-20T17:48:29.686 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b50af0700 1 -- start start 2023-07-20T17:48:29.686 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b50af0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c0ff720 0x7f3b4c1036b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:29.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b50af0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b4c100910 0x7f3b4c101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:29.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b50af0700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4c1012b0 0x7f3b4c102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:29.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b50af0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c19a330 con 0x7f3b4c100910 2023-07-20T17:48:29.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b50af0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c102770 con 0x7f3b4c0ff720 2023-07-20T17:48:29.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b50af0700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c102930 con 0x7f3b4c1012b0 2023-07-20T17:48:29.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b49d9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b4c100910 0x7f3b4c101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:29.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b49d9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b4c100910 0x7f3b4c101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36248/0 (socket says 172.21.15.49:36248) 2023-07-20T17:48:29.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b49d9b700 1 -- 172.21.15.49:0/284675603 learned_addr learned my addr 172.21.15.49:0/284675603 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:29.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.681+0000 7f3b4a59c700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c0ff720 0x7f3b4c1036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:29.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b4ad9d700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4c1012b0 0x7f3b4c102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:29.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b49d9b700 1 -- 172.21.15.49:0/284675603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4c1012b0 msgr2=0x7f3b4c102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:29.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b49d9b700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4c1012b0 0x7f3b4c102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b49d9b700 1 -- 172.21.15.49:0/284675603 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c0ff720 msgr2=0x7f3b4c1036b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:29.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b49d9b700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c0ff720 0x7f3b4c1036b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b49d9b700 1 -- 172.21.15.49:0/284675603 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c0095f0 con 0x7f3b4c100910 2023-07-20T17:48:29.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b49d9b700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b4c100910 0x7f3b4c101cf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3b4000e9a0 tx=0x7f3b4000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:29.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b3b7fe700 1 -- 172.21.15.49:0/284675603 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b40009790 con 0x7f3b4c100910 2023-07-20T17:48:29.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b3b7fe700 1 -- 172.21.15.49:0/284675603 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b40010430 con 0x7f3b4c100910 2023-07-20T17:48:29.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b50af0700 1 -- 172.21.15.49:0/284675603 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4c1a4ce0 con 0x7f3b4c100910 2023-07-20T17:48:29.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b3b7fe700 1 -- 172.21.15.49:0/284675603 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b40018680 con 0x7f3b4c100910 2023-07-20T17:48:29.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.682+0000 7f3b50af0700 1 -- 172.21.15.49:0/284675603 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4c1a5100 con 0x7f3b4c100910 2023-07-20T17:48:29.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.683+0000 7f3b50af0700 1 -- 172.21.15.49:0/284675603 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b4c10af30 con 0x7f3b4c100910 2023-07-20T17:48:29.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.684+0000 7f3b3b7fe700 1 -- 172.21.15.49:0/284675603 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3b400105a0 con 0x7f3b4c100910 2023-07-20T17:48:29.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.685+0000 7f3b3b7fe700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f3b340a1150 0x7f3b340a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:29.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.685+0000 7f3b3b7fe700 1 -- 172.21.15.49:0/284675603 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3b40014070 con 0x7f3b4c100910 2023-07-20T17:48:29.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.685+0000 7f3b4a59c700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f3b340a1150 0x7f3b340a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:29.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.685+0000 7f3b4a59c700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f3b340a1150 0x7f3b340a3610 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009990 tx=0x7f3b3c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:29.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.688+0000 7f3b3b7fe700 1 -- 172.21.15.49:0/284675603 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3b40096400 con 0x7f3b4c100910 2023-07-20T17:48:29.935 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.932+0000 7f3b50af0700 1 -- 172.21.15.49:0/284675603 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b4c10b0f0 con 0x7f3b4c100910 2023-07-20T17:48:29.935 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.932+0000 7f3b3b7fe700 1 -- 172.21.15.49:0/284675603 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3528 (secure 0 0 0) 0x7f3b40026de0 con 0x7f3b4c100910 2023-07-20T17:48:29.936 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:29.937 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1572141770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1572141770},{"type":"v1","addr":"172.21.15.49:6835","nonce":1572141770}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/4279110827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":4279110827},{"type":"v1","addr":"172.21.15.49:6837","nonce":4279110827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/1293946253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1293946253},{"type":"v1","addr":"172.21.15.97:6833","nonce":1293946253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2111036634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2111036634},{"type":"v1","addr":"172.21.15.79:6833","nonce":2111036634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-20T17:48:29.938 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.934+0000 7f3b50af0700 1 -- 172.21.15.49:0/284675603 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f3b340a1150 msgr2=0x7f3b340a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:29.939 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.934+0000 7f3b50af0700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f3b340a1150 0x7f3b340a3610 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009990 tx=0x7f3b3c005fb0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.939 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.934+0000 7f3b50af0700 1 -- 172.21.15.49:0/284675603 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b4c100910 msgr2=0x7f3b4c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:29.939 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.934+0000 7f3b50af0700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b4c100910 0x7f3b4c101cf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3b4000e9a0 tx=0x7f3b4000ecb0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.939 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.934+0000 7f3b50af0700 1 -- 172.21.15.49:0/284675603 shutdown_connections 2023-07-20T17:48:29.940 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.934+0000 7f3b50af0700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f3b340a1150 0x7f3b340a3610 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.940 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.934+0000 7f3b50af0700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b4c1012b0 0x7f3b4c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.940 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.934+0000 7f3b50af0700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b4c0ff720 0x7f3b4c1036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.934+0000 7f3b50af0700 1 --2- 172.21.15.49:0/284675603 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b4c100910 0x7f3b4c101cf0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:29.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.934+0000 7f3b50af0700 1 -- 172.21.15.49:0/284675603 >> 172.21.15.49:0/284675603 conn(0x7f3b4c0facb0 msgr2=0x7f3b4c067ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:29.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.934+0000 7f3b50af0700 1 -- 172.21.15.49:0/284675603 shutdown_connections 2023-07-20T17:48:29.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:29.935+0000 7f3b50af0700 1 -- 172.21.15.49:0/284675603 wait complete. 2023-07-20T17:48:29.942 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 5 2023-07-20T17:48:30.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:30.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:30.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:30.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:30.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:30.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:30.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:30.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: Deploying daemon mds.f on smithi097 2023-07-20T17:48:30.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/284675603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:30.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.79:6834/3824893609,v1:172.21.15.79:6835/3824893609] up:boot 2023-07-20T17:48:30.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: fsmap 5 up:standby 2023-07-20T17:48:30.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:30 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T17:48:30.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:30.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:30.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:30.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:30.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:30.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:30.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:30.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: Deploying daemon mds.f on smithi097 2023-07-20T17:48:30.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/284675603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:30.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.79:6834/3824893609,v1:172.21.15.79:6835/3824893609] up:boot 2023-07-20T17:48:30.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: fsmap 5 up:standby 2023-07-20T17:48:30.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:30 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T17:48:30.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:30.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:30.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:30.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:30.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:30.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:30.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:30.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: Deploying daemon mds.f on smithi097 2023-07-20T17:48:30.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/284675603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:30.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.79:6834/3824893609,v1:172.21.15.79:6835/3824893609] up:boot 2023-07-20T17:48:30.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: fsmap 5 up:standby 2023-07-20T17:48:30.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:30 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T17:48:31.018 INFO:journalctl@ceph.mds.f.smithi097.stdout:Jul 20 17:48:30 smithi097 systemd[1]: Starting Ceph mds.f for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:31.518 INFO:journalctl@ceph.mds.f.smithi097.stdout:Jul 20 17:48:31 smithi097 podman[123850]: 2023-07-20T17:48:31.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:32.018 INFO:journalctl@ceph.mds.f.smithi097.stdout:Jul 20 17:48:31 smithi097 ceph-mds[123890]: -- 172.21.15.97:0/1748234764 <== mon.1 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5564b58dd1e0 con 0x5564b58b2800 2023-07-20T17:48:32.018 INFO:journalctl@ceph.mds.f.smithi097.stdout:Jul 20 17:48:31 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-f[123866]: starting mds.f at 2023-07-20T17:48:32.018 INFO:journalctl@ceph.mds.f.smithi097.stdout:Jul 20 17:48:31 smithi097 bash[123850]: f65ef14cd0ec00183f7a581a918f48b2b69c63aff68a716ec6c19d3dafd827e3 2023-07-20T17:48:32.018 INFO:journalctl@ceph.mds.f.smithi097.stdout:Jul 20 17:48:31 smithi097 systemd[1]: Started Ceph mds.f for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:32.093 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.091+0000 7f969967e700 1 -- 172.21.15.49:0/502856387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9694101290 msgr2=0x7f9694107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:32.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.091+0000 7f969967e700 1 --2- 172.21.15.49:0/502856387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9694101290 0x7f9694107b40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f968c009f90 tx=0x7f968c00b330 comp rx=0 tx=0).stop 2023-07-20T17:48:32.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.091+0000 7f969967e700 1 -- 172.21.15.49:0/502856387 shutdown_connections 2023-07-20T17:48:32.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.091+0000 7f969967e700 1 --2- 172.21.15.49:0/502856387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9694101290 0x7f9694107b40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:32.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.091+0000 7f969967e700 1 --2- 172.21.15.49:0/502856387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96940ff700 0x7f96940ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:32.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.091+0000 7f969967e700 1 --2- 172.21.15.49:0/502856387 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96941008f0 0x7f9694100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:32.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.091+0000 7f969967e700 1 -- 172.21.15.49:0/502856387 >> 172.21.15.49:0/502856387 conn(0x7f96940fac70 msgr2=0x7f96940fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:32.095 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.091+0000 7f969967e700 1 -- 172.21.15.49:0/502856387 shutdown_connections 2023-07-20T17:48:32.095 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.091+0000 7f969967e700 1 -- 172.21.15.49:0/502856387 wait complete. 2023-07-20T17:48:32.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.091+0000 7f969967e700 1 Processor -- start 2023-07-20T17:48:32.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f969967e700 1 -- start start 2023-07-20T17:48:32.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f969967e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96940ff700 0x7f96941965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:32.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f969967e700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96941008f0 0x7f9694196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:32.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f969967e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9694101290 0x7f969419db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:32.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f969967e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969419e160 con 0x7f96940ff700 2023-07-20T17:48:32.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f969967e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969419e2a0 con 0x7f9694101290 2023-07-20T17:48:32.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f969967e700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969419e430 con 0x7f96941008f0 2023-07-20T17:48:32.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f9692ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96940ff700 0x7f96941965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:32.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f9692ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96940ff700 0x7f96941965e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36260/0 (socket says 172.21.15.49:36260) 2023-07-20T17:48:32.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f9692ffd700 1 -- 172.21.15.49:0/1730344991 learned_addr learned my addr 172.21.15.49:0/1730344991 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:32.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f96927fc700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96941008f0 0x7f9694196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:32.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f96937fe700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9694101290 0x7f969419db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:32.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f9692ffd700 1 -- 172.21.15.49:0/1730344991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96941008f0 msgr2=0x7f9694196b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:32.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f9692ffd700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96941008f0 0x7f9694196b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:32.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f9692ffd700 1 -- 172.21.15.49:0/1730344991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9694101290 msgr2=0x7f969419db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:32.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f9692ffd700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9694101290 0x7f969419db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:32.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f9692ffd700 1 -- 172.21.15.49:0/1730344991 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96880095f0 con 0x7f96940ff700 2023-07-20T17:48:32.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.092+0000 7f9692ffd700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96940ff700 0x7f96941965e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f968400d980 tx=0x7f968400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:32.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.093+0000 7f967bfff700 1 -- 172.21.15.49:0/1730344991 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968400f890 con 0x7f96940ff700 2023-07-20T17:48:32.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.093+0000 7f969967e700 1 -- 172.21.15.49:0/1730344991 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968c009c40 con 0x7f96940ff700 2023-07-20T17:48:32.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.093+0000 7f967bfff700 1 -- 172.21.15.49:0/1730344991 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9684010460 con 0x7f96940ff700 2023-07-20T17:48:32.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.093+0000 7f967bfff700 1 -- 172.21.15.49:0/1730344991 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96840195c0 con 0x7f96940ff700 2023-07-20T17:48:32.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.093+0000 7f969967e700 1 -- 172.21.15.49:0/1730344991 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969419eae0 con 0x7f96940ff700 2023-07-20T17:48:32.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.094+0000 7f969967e700 1 -- 172.21.15.49:0/1730344991 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9694107790 con 0x7f96940ff700 2023-07-20T17:48:32.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.095+0000 7f967bfff700 1 -- 172.21.15.49:0/1730344991 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f96840105d0 con 0x7f96940ff700 2023-07-20T17:48:32.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.096+0000 7f967bfff700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f967c0a1150 0x7f967c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:32.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.096+0000 7f96927fc700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f967c0a1150 0x7f967c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:32.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.096+0000 7f967bfff700 1 -- 172.21.15.49:0/1730344991 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f96840cc710 con 0x7f96940ff700 2023-07-20T17:48:32.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.097+0000 7f96927fc700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f967c0a1150 0x7f967c0a3610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9688005fd0 tx=0x7f9688009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:32.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.099+0000 7f967bfff700 1 -- 172.21.15.49:0/1730344991 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9684096300 con 0x7f96940ff700 2023-07-20T17:48:32.348 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.346+0000 7f969967e700 1 -- 172.21.15.49:0/1730344991 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f969402ce00 con 0x7f96940ff700 2023-07-20T17:48:32.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.346+0000 7f967bfff700 1 -- 172.21.15.49:0/1730344991 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5042 (secure 0 0 0) 0x7f9684099920 con 0x7f96940ff700 2023-07-20T17:48:32.349 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:32.349 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1572141770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1572141770},{"type":"v1","addr":"172.21.15.49:6835","nonce":1572141770}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/4279110827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":4279110827},{"type":"v1","addr":"172.21.15.49:6837","nonce":4279110827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/1293946253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1293946253},{"type":"v1","addr":"172.21.15.97:6833","nonce":1293946253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2111036634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2111036634},{"type":"v1","addr":"172.21.15.79:6833","nonce":2111036634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/3824893609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":3824893609},{"type":"v1","addr":"172.21.15.79:6835","nonce":3824893609}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24484,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1698011127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1698011127},{"type":"v1","addr":"172.21.15.97:6835","nonce":1698011127}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-07-20T17:48:32.351 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 -- 172.21.15.49:0/1730344991 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f967c0a1150 msgr2=0x7f967c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:32.351 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f967c0a1150 0x7f967c0a3610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9688005fd0 tx=0x7f9688009310 comp rx=0 tx=0).stop 2023-07-20T17:48:32.351 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 -- 172.21.15.49:0/1730344991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96940ff700 msgr2=0x7f96941965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:32.351 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96940ff700 0x7f96941965e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f968400d980 tx=0x7f968400dc90 comp rx=0 tx=0).stop 2023-07-20T17:48:32.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 -- 172.21.15.49:0/1730344991 shutdown_connections 2023-07-20T17:48:32.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f967c0a1150 0x7f967c0a3610 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:32.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9694101290 0x7f969419db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:32.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96940ff700 0x7f96941965e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:32.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 --2- 172.21.15.49:0/1730344991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96941008f0 0x7f9694196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:32.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 -- 172.21.15.49:0/1730344991 >> 172.21.15.49:0/1730344991 conn(0x7f96940fac70 msgr2=0x7f96940fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:32.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 -- 172.21.15.49:0/1730344991 shutdown_connections 2023-07-20T17:48:32.353 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:32.348+0000 7f969967e700 1 -- 172.21.15.49:0/1730344991 wait complete. 2023-07-20T17:48:32.353 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 7 2023-07-20T17:48:32.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:32.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:32.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:32.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:32.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:32.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:32.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:32.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: Deploying daemon mds.c on smithi097 2023-07-20T17:48:32.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.97:6834/1698011127,v1:172.21.15.97:6835/1698011127] up:boot 2023-07-20T17:48:32.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: fsmap 6 up:standby 2023-07-20T17:48:32.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T17:48:32.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:32 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1730344991' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:33.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:33.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:33.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:33.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:33.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:33.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:33.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:33.021 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: Deploying daemon mds.c on smithi097 2023-07-20T17:48:33.021 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.97:6834/1698011127,v1:172.21.15.97:6835/1698011127] up:boot 2023-07-20T17:48:33.021 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: fsmap 6 up:standby 2023-07-20T17:48:33.022 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T17:48:33.022 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:32 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1730344991' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:33.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:33.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:33.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:33.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:33.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:33.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:33.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:33.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: Deploying daemon mds.c on smithi097 2023-07-20T17:48:33.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.97:6834/1698011127,v1:172.21.15.97:6835/1698011127] up:boot 2023-07-20T17:48:33.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: fsmap 6 up:standby 2023-07-20T17:48:33.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T17:48:33.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:32 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1730344991' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:33.350 INFO:journalctl@ceph.mds.c.smithi097.stdout:Jul 20 17:48:33 smithi097 systemd[1]: Starting Ceph mds.c for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:33.935 INFO:journalctl@ceph.mds.c.smithi097.stdout:Jul 20 17:48:33 smithi097 podman[124188]: 2023-07-20T17:48:34.268 INFO:journalctl@ceph.mds.c.smithi097.stdout:Jul 20 17:48:33 smithi097 ceph-mds[124228]: -- 172.21.15.97:0/4133632445 <== mon.1 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d5e28d71e0 con 0x55d5e28acc00 2023-07-20T17:48:34.268 INFO:journalctl@ceph.mds.c.smithi097.stdout:Jul 20 17:48:33 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-c[124204]: starting mds.c at 2023-07-20T17:48:34.268 INFO:journalctl@ceph.mds.c.smithi097.stdout:Jul 20 17:48:33 smithi097 bash[124188]: 40b08eb819f95027720b2450802d420bf8669b66cf9469667be4de8de7d34aad 2023-07-20T17:48:34.269 INFO:journalctl@ceph.mds.c.smithi097.stdout:Jul 20 17:48:34 smithi097 systemd[1]: Started Ceph mds.c for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:34.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:34.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.501+0000 7ff39ee20700 1 -- 172.21.15.49:0/838362478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff398100920 msgr2=0x7ff398100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:34.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.501+0000 7ff39ee20700 1 --2- 172.21.15.49:0/838362478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff398100920 0x7ff398100d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff3840099c0 tx=0x7ff384009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:34.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.501+0000 7ff39ee20700 1 -- 172.21.15.49:0/838362478 shutdown_connections 2023-07-20T17:48:34.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.501+0000 7ff39ee20700 1 --2- 172.21.15.49:0/838362478 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff3981012c0 0x7ff398107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:34.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.501+0000 7ff39ee20700 1 --2- 172.21.15.49:0/838362478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3980ff730 0x7ff3980ffb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:34.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.501+0000 7ff39ee20700 1 --2- 172.21.15.49:0/838362478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff398100920 0x7ff398100d80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:34.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.501+0000 7ff39ee20700 1 -- 172.21.15.49:0/838362478 >> 172.21.15.49:0/838362478 conn(0x7ff3980faca0 msgr2=0x7ff3980fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:34.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39ee20700 1 -- 172.21.15.49:0/838362478 shutdown_connections 2023-07-20T17:48:34.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39ee20700 1 -- 172.21.15.49:0/838362478 wait complete. 2023-07-20T17:48:34.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39ee20700 1 Processor -- start 2023-07-20T17:48:34.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39ee20700 1 -- start start 2023-07-20T17:48:34.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39ee20700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3980ff730 0x7ff398069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:34.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39ee20700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff398100920 0x7ff39819b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:34.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39ee20700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3981012c0 0x7ff39819d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:34.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39ee20700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39819dfe0 con 0x7ff3980ff730 2023-07-20T17:48:34.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39ee20700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39819e150 con 0x7ff3981012c0 2023-07-20T17:48:34.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39ee20700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39819e310 con 0x7ff398100920 2023-07-20T17:48:34.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39cbbc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3980ff730 0x7ff398069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:34.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39cbbc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3980ff730 0x7ff398069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36274/0 (socket says 172.21.15.49:36274) 2023-07-20T17:48:34.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39cbbc700 1 -- 172.21.15.49:0/3804878291 learned_addr learned my addr 172.21.15.49:0/3804878291 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:34.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.502+0000 7ff39d3bd700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3981012c0 0x7ff39819d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:34.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff397fff700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff398100920 0x7ff39819b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:34.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff39cbbc700 1 -- 172.21.15.49:0/3804878291 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff398100920 msgr2=0x7ff39819b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:34.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff39cbbc700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff398100920 0x7ff39819b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:34.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff39cbbc700 1 -- 172.21.15.49:0/3804878291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3981012c0 msgr2=0x7ff39819d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:34.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff39cbbc700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3981012c0 0x7ff39819d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:34.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff39cbbc700 1 -- 172.21.15.49:0/3804878291 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3840095f0 con 0x7ff3980ff730 2023-07-20T17:48:34.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff397fff700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff398100920 0x7ff39819b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T17:48:34.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff39cbbc700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3980ff730 0x7ff398069390 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00e970 tx=0x7ff38c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:34.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff395ffb700 1 -- 172.21.15.49:0/3804878291 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c009790 con 0x7ff3980ff730 2023-07-20T17:48:34.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff395ffb700 1 -- 172.21.15.49:0/3804878291 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff38c00eec0 con 0x7ff3980ff730 2023-07-20T17:48:34.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff39ee20700 1 -- 172.21.15.49:0/3804878291 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39819e5f0 con 0x7ff3980ff730 2023-07-20T17:48:34.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff395ffb700 1 -- 172.21.15.49:0/3804878291 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c003cf0 con 0x7ff3980ff730 2023-07-20T17:48:34.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.503+0000 7ff39ee20700 1 -- 172.21.15.49:0/3804878291 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39819eb50 con 0x7ff3980ff730 2023-07-20T17:48:34.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.504+0000 7ff39ee20700 1 -- 172.21.15.49:0/3804878291 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff398106610 con 0x7ff3980ff730 2023-07-20T17:48:34.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.505+0000 7ff395ffb700 1 -- 172.21.15.49:0/3804878291 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff38c00ccd0 con 0x7ff3980ff730 2023-07-20T17:48:34.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.506+0000 7ff395ffb700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff3880a11d0 0x7ff3880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:34.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.506+0000 7ff397fff700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff3880a11d0 0x7ff3880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:34.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.506+0000 7ff395ffb700 1 -- 172.21.15.49:0/3804878291 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff38c014070 con 0x7ff3980ff730 2023-07-20T17:48:34.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.507+0000 7ff397fff700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff3880a11d0 0x7ff3880a3690 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff384006010 tx=0x7ff384005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:34.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.511+0000 7ff395ffb700 1 -- 172.21.15.49:0/3804878291 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff38c096210 con 0x7ff3980ff730 2023-07-20T17:48:34.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.757+0000 7ff39ee20700 1 -- 172.21.15.49:0/3804878291 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff39802c970 con 0x7ff3980ff730 2023-07-20T17:48:34.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.757+0000 7ff395ffb700 1 -- 172.21.15.49:0/3804878291 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5796 (secure 0 0 0) 0x7ff38c099830 con 0x7ff3980ff730 2023-07-20T17:48:34.761 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:34.761 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1572141770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1572141770},{"type":"v1","addr":"172.21.15.49:6835","nonce":1572141770}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/4279110827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":4279110827},{"type":"v1","addr":"172.21.15.49:6837","nonce":4279110827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/1293946253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1293946253},{"type":"v1","addr":"172.21.15.97:6833","nonce":1293946253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2111036634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2111036634},{"type":"v1","addr":"172.21.15.79:6833","nonce":2111036634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/3824893609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":3824893609},{"type":"v1","addr":"172.21.15.79:6835","nonce":3824893609}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24484,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1698011127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1698011127},{"type":"v1","addr":"172.21.15.97:6835","nonce":1698011127}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/752441359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":752441359},{"type":"v1","addr":"172.21.15.97:6837","nonce":752441359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-20T17:48:34.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.759+0000 7ff39ee20700 1 -- 172.21.15.49:0/3804878291 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff3880a11d0 msgr2=0x7ff3880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:34.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.759+0000 7ff39ee20700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff3880a11d0 0x7ff3880a3690 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff384006010 tx=0x7ff384005e80 comp rx=0 tx=0).stop 2023-07-20T17:48:34.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.760+0000 7ff39ee20700 1 -- 172.21.15.49:0/3804878291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3980ff730 msgr2=0x7ff398069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:34.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.760+0000 7ff39ee20700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3980ff730 0x7ff398069390 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00e970 tx=0x7ff38c00ec80 comp rx=0 tx=0).stop 2023-07-20T17:48:34.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.760+0000 7ff39ee20700 1 -- 172.21.15.49:0/3804878291 shutdown_connections 2023-07-20T17:48:34.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.760+0000 7ff39ee20700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff3880a11d0 0x7ff3880a3690 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:34.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.760+0000 7ff39ee20700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3981012c0 0x7ff39819d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:34.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.760+0000 7ff39ee20700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3980ff730 0x7ff398069390 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:34.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.760+0000 7ff39ee20700 1 --2- 172.21.15.49:0/3804878291 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff398100920 0x7ff39819b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:34.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.760+0000 7ff39ee20700 1 -- 172.21.15.49:0/3804878291 >> 172.21.15.49:0/3804878291 conn(0x7ff3980faca0 msgr2=0x7ff3980fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:34.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.760+0000 7ff39ee20700 1 -- 172.21.15.49:0/3804878291 shutdown_connections 2023-07-20T17:48:34.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:34.760+0000 7ff39ee20700 1 -- 172.21.15.49:0/3804878291 wait complete. 2023-07-20T17:48:34.764 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 8 2023-07-20T17:48:34.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:34 smithi049 ceph-mon[96530]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:34.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:34.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:34.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:34.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:34.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:34.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:34 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:35.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:34 smithi097 ceph-mon[98398]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:35.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:35.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:35.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:35.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:35.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:35.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:34 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:35.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:34 smithi079 ceph-mon[98036]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:35.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:35.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:35.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:35.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:35.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:35.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:34 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:35.727 INFO:journalctl@ceph.mds.h.smithi079.stdout:Jul 20 17:48:35 smithi079 systemd[1]: Starting Ceph mds.h for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:35.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:35 smithi049 ceph-mon[96530]: Deploying daemon mds.h on smithi079 2023-07-20T17:48:35.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:35 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.97:6836/752441359,v1:172.21.15.97:6837/752441359] up:boot 2023-07-20T17:48:35.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:35 smithi049 ceph-mon[96530]: fsmap 7 up:standby 2023-07-20T17:48:35.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:35 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T17:48:35.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:35 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3804878291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:36.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:35 smithi097 ceph-mon[98398]: Deploying daemon mds.h on smithi079 2023-07-20T17:48:36.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:35 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.97:6836/752441359,v1:172.21.15.97:6837/752441359] up:boot 2023-07-20T17:48:36.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:35 smithi097 ceph-mon[98398]: fsmap 7 up:standby 2023-07-20T17:48:36.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:35 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T17:48:36.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:35 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3804878291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:36.044 INFO:journalctl@ceph.mds.h.smithi079.stdout:Jul 20 17:48:35 smithi079 podman[122976]: 2023-07-20T17:48:36.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:35 smithi079 ceph-mon[98036]: Deploying daemon mds.h on smithi079 2023-07-20T17:48:36.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:35 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.97:6836/752441359,v1:172.21.15.97:6837/752441359] up:boot 2023-07-20T17:48:36.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:35 smithi079 ceph-mon[98036]: fsmap 7 up:standby 2023-07-20T17:48:36.044 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:35 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T17:48:36.045 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:35 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3804878291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:36.567 INFO:journalctl@ceph.mds.h.smithi079.stdout:Jul 20 17:48:36 smithi079 ceph-mds[123016]: -- 172.21.15.79:0/1265238877 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561d0e8851e0 con 0x561d0e85b000 2023-07-20T17:48:36.568 INFO:journalctl@ceph.mds.h.smithi079.stdout:Jul 20 17:48:36 smithi079 bash[122976]: 3f7531b1ed9d2a32b16302371c42da86d5b262fd29d3edea298dcd25b15377e1 2023-07-20T17:48:36.568 INFO:journalctl@ceph.mds.h.smithi079.stdout:Jul 20 17:48:36 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-h[122992]: starting mds.h at 2023-07-20T17:48:36.568 INFO:journalctl@ceph.mds.h.smithi079.stdout:Jul 20 17:48:36 smithi079 systemd[1]: Started Ceph mds.h for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:36.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:36.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.911+0000 7f25dba82700 1 -- 172.21.15.49:0/3548870181 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25d4100910 msgr2=0x7f25d4100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:36.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.911+0000 7f25dba82700 1 --2- 172.21.15.49:0/3548870181 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25d4100910 0x7f25d4100d70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f25c00099c0 tx=0x7f25c0009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:36.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.911+0000 7f25dba82700 1 -- 172.21.15.49:0/3548870181 shutdown_connections 2023-07-20T17:48:36.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.911+0000 7f25dba82700 1 --2- 172.21.15.49:0/3548870181 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25d40ff720 0x7f25d40ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:36.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.911+0000 7f25dba82700 1 --2- 172.21.15.49:0/3548870181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25d41012b0 0x7f25d4107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:36.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.911+0000 7f25dba82700 1 --2- 172.21.15.49:0/3548870181 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25d4100910 0x7f25d4100d70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:36.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.911+0000 7f25dba82700 1 -- 172.21.15.49:0/3548870181 >> 172.21.15.49:0/3548870181 conn(0x7f25d40facb0 msgr2=0x7f25d40fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:36.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.911+0000 7f25dba82700 1 -- 172.21.15.49:0/3548870181 shutdown_connections 2023-07-20T17:48:36.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.911+0000 7f25dba82700 1 -- 172.21.15.49:0/3548870181 wait complete. 2023-07-20T17:48:36.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25dba82700 1 Processor -- start 2023-07-20T17:48:36.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25dba82700 1 -- start start 2023-07-20T17:48:36.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25dba82700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25d40ff720 0x7f25d4195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:36.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25dba82700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25d4100910 0x7f25d4195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:36.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25dba82700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25d41012b0 0x7f25d419ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:36.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25dba82700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d419cfd0 con 0x7f25d4100910 2023-07-20T17:48:36.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25dba82700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d419d110 con 0x7f25d41012b0 2023-07-20T17:48:36.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25dba82700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d419d2a0 con 0x7f25d40ff720 2023-07-20T17:48:36.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25d901d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25d4100910 0x7f25d4195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:36.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25d901d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25d4100910 0x7f25d4195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36290/0 (socket says 172.21.15.49:36290) 2023-07-20T17:48:36.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25d901d700 1 -- 172.21.15.49:0/2070982510 learned_addr learned my addr 172.21.15.49:0/2070982510 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:36.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25da01f700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25d41012b0 0x7f25d419ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:36.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25d981e700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25d40ff720 0x7f25d4195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:36.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25d901d700 1 -- 172.21.15.49:0/2070982510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25d40ff720 msgr2=0x7f25d4195450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:36.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.912+0000 7f25d901d700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25d40ff720 0x7f25d4195450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:36.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.913+0000 7f25d901d700 1 -- 172.21.15.49:0/2070982510 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25d41012b0 msgr2=0x7f25d419ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:36.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.913+0000 7f25d901d700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25d41012b0 0x7f25d419ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:36.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.913+0000 7f25d901d700 1 -- 172.21.15.49:0/2070982510 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00095f0 con 0x7f25d4100910 2023-07-20T17:48:36.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.913+0000 7f25d981e700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25d40ff720 0x7f25d4195450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:48:36.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.913+0000 7f25d901d700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25d4100910 0x7f25d4195990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009990 tx=0x7f25c000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:36.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.913+0000 7f25d2ffd700 1 -- 172.21.15.49:0/2070982510 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c001d070 con 0x7f25d4100910 2023-07-20T17:48:36.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.913+0000 7f25dba82700 1 -- 172.21.15.49:0/2070982510 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d419d520 con 0x7f25d4100910 2023-07-20T17:48:36.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.913+0000 7f25d2ffd700 1 -- 172.21.15.49:0/2070982510 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25c00054f0 con 0x7f25d4100910 2023-07-20T17:48:36.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.913+0000 7f25d2ffd700 1 -- 172.21.15.49:0/2070982510 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0004b00 con 0x7f25d4100910 2023-07-20T17:48:36.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.913+0000 7f25dba82700 1 -- 172.21.15.49:0/2070982510 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d4067a40 con 0x7f25d4100910 2023-07-20T17:48:36.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.914+0000 7f25dba82700 1 -- 172.21.15.49:0/2070982510 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d410afb0 con 0x7f25d4100910 2023-07-20T17:48:36.922 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.915+0000 7f25d2ffd700 1 -- 172.21.15.49:0/2070982510 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f25c000fb80 con 0x7f25d4100910 2023-07-20T17:48:36.922 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.916+0000 7f25d2ffd700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f25c40a3360 0x7f25c40a5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:36.922 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.916+0000 7f25d981e700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f25c40a3360 0x7f25c40a5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:36.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.916+0000 7f25d2ffd700 1 -- 172.21.15.49:0/2070982510 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f25c00cd660 con 0x7f25d4100910 2023-07-20T17:48:36.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.916+0000 7f25d981e700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f25c40a3360 0x7f25c40a5820 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f25d4100770 tx=0x7f25c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:36.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:36.919+0000 7f25d2ffd700 1 -- 172.21.15.49:0/2070982510 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f25c00972e0 con 0x7f25d4100910 2023-07-20T17:48:36.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:36 smithi049 ceph-mon[96530]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:36.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:36.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:36.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:36.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:36.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:36.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:36 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:37.000 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:36 smithi097 ceph-mon[98398]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:37.000 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:37.001 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:37.001 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:37.001 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:37.001 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:37.001 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:36 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:37.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:36 smithi079 ceph-mon[98036]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:37.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:37.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:37.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:37.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:37.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:37.070 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:36 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:37.172 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.169+0000 7f25dba82700 1 -- 172.21.15.49:0/2070982510 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25d4068850 con 0x7f25d4100910 2023-07-20T17:48:37.173 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.170+0000 7f25d2ffd700 1 -- 172.21.15.49:0/2070982510 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6553 (secure 0 0 0) 0x7f25c009a900 con 0x7f25d4100910 2023-07-20T17:48:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:37.174 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1572141770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1572141770},{"type":"v1","addr":"172.21.15.49:6835","nonce":1572141770}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/4279110827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":4279110827},{"type":"v1","addr":"172.21.15.49:6837","nonce":4279110827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/1293946253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1293946253},{"type":"v1","addr":"172.21.15.97:6833","nonce":1293946253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2111036634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2111036634},{"type":"v1","addr":"172.21.15.79:6833","nonce":2111036634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/3824893609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":3824893609},{"type":"v1","addr":"172.21.15.79:6835","nonce":3824893609}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24484,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1698011127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1698011127},{"type":"v1","addr":"172.21.15.97:6835","nonce":1698011127}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/752441359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":752441359},{"type":"v1","addr":"172.21.15.97:6837","nonce":752441359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3288313742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3288313742},{"type":"v1","addr":"172.21.15.79:6837","nonce":3288313742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-20T17:48:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 -- 172.21.15.49:0/2070982510 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f25c40a3360 msgr2=0x7f25c40a5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f25c40a3360 0x7f25c40a5820 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f25d4100770 tx=0x7f25c8009260 comp rx=0 tx=0).stop 2023-07-20T17:48:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 -- 172.21.15.49:0/2070982510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25d4100910 msgr2=0x7f25d4195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:37.177 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25d4100910 0x7f25d4195990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009990 tx=0x7f25c000f690 comp rx=0 tx=0).stop 2023-07-20T17:48:37.177 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 -- 172.21.15.49:0/2070982510 shutdown_connections 2023-07-20T17:48:37.177 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f25c40a3360 0x7f25c40a5820 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:37.177 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25d40ff720 0x7f25d4195450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:37.178 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25d41012b0 0x7f25d419ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:37.178 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 --2- 172.21.15.49:0/2070982510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25d4100910 0x7f25d4195990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:37.178 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 -- 172.21.15.49:0/2070982510 >> 172.21.15.49:0/2070982510 conn(0x7f25d40facb0 msgr2=0x7f25d40fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:37.179 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 -- 172.21.15.49:0/2070982510 shutdown_connections 2023-07-20T17:48:37.179 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:37.172+0000 7f25dba82700 1 -- 172.21.15.49:0/2070982510 wait complete. 2023-07-20T17:48:37.179 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 9 2023-07-20T17:48:37.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:37 smithi049 ceph-mon[96530]: Deploying daemon mds.i on smithi097 2023-07-20T17:48:37.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:37 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.79:6836/3288313742,v1:172.21.15.79:6837/3288313742] up:boot 2023-07-20T17:48:37.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:37 smithi049 ceph-mon[96530]: fsmap 8 up:standby 2023-07-20T17:48:37.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:37 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T17:48:37.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:37 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2070982510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:38.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:37 smithi097 ceph-mon[98398]: Deploying daemon mds.i on smithi097 2023-07-20T17:48:38.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:37 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.79:6836/3288313742,v1:172.21.15.79:6837/3288313742] up:boot 2023-07-20T17:48:38.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:37 smithi097 ceph-mon[98398]: fsmap 8 up:standby 2023-07-20T17:48:38.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:37 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T17:48:38.020 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:37 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/2070982510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:38.021 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:48:37 smithi097 systemd[1]: Starting Ceph mds.i for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:38.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:37 smithi079 ceph-mon[98036]: Deploying daemon mds.i on smithi097 2023-07-20T17:48:38.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:37 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.79:6836/3288313742,v1:172.21.15.79:6837/3288313742] up:boot 2023-07-20T17:48:38.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:37 smithi079 ceph-mon[98036]: fsmap 8 up:standby 2023-07-20T17:48:38.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:37 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T17:48:38.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:37 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/2070982510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:38.584 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:48:38 smithi097 podman[124528]: 2023-07-20T17:48:39.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:38 smithi097 ceph-mon[98398]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:39.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:39.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:38 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:39.018 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:48:38 smithi097 ceph-mds[124568]: -- 172.21.15.97:0/3437879236 <== mon.1 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c4fdcb31e0 con 0x55c4fdc89000 2023-07-20T17:48:39.019 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:48:38 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-i[124544]: starting mds.i at 2023-07-20T17:48:39.019 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:48:38 smithi097 bash[124528]: 0028bd080dafb2e22b31a82f86b75c5f008f627ddd6d1ab866ac6b0feb858ca3 2023-07-20T17:48:39.019 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:48:38 smithi097 systemd[1]: Started Ceph mds.i for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:39.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:38 smithi079 ceph-mon[98036]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:39.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:39.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:38 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:39.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:39.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:38 smithi049 ceph-mon[96530]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:39.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:39.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:38 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:39.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.317+0000 7ff775ec0700 1 -- 172.21.15.49:0/844489031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff7701012b0 msgr2=0x7ff770107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:39.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.317+0000 7ff775ec0700 1 --2- 172.21.15.49:0/844489031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff7701012b0 0x7ff770107b60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff76800b0d0 tx=0x7ff76800b3e0 comp rx=0 tx=0).stop 2023-07-20T17:48:39.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.317+0000 7ff775ec0700 1 -- 172.21.15.49:0/844489031 shutdown_connections 2023-07-20T17:48:39.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.317+0000 7ff775ec0700 1 --2- 172.21.15.49:0/844489031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7700ff720 0x7ff7700ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:39.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.317+0000 7ff775ec0700 1 --2- 172.21.15.49:0/844489031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff7701012b0 0x7ff770107b60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:39.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.317+0000 7ff775ec0700 1 --2- 172.21.15.49:0/844489031 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff770100910 0x7ff770100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:39.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.317+0000 7ff775ec0700 1 -- 172.21.15.49:0/844489031 >> 172.21.15.49:0/844489031 conn(0x7ff7700facb0 msgr2=0x7ff7700fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:39.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.317+0000 7ff775ec0700 1 -- 172.21.15.49:0/844489031 shutdown_connections 2023-07-20T17:48:39.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.318+0000 7ff775ec0700 1 -- 172.21.15.49:0/844489031 wait complete. 2023-07-20T17:48:39.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.318+0000 7ff775ec0700 1 Processor -- start 2023-07-20T17:48:39.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff775ec0700 1 -- start start 2023-07-20T17:48:39.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff775ec0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff7700ff720 0x7ff770068610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:39.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff775ec0700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff770100910 0x7ff77019b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:39.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff775ec0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7701012b0 0x7ff77019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:39.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff775ec0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff770195070 con 0x7ff7700ff720 2023-07-20T17:48:39.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff775ec0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7701951e0 con 0x7ff7701012b0 2023-07-20T17:48:39.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff775ec0700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7701953a0 con 0x7ff770100910 2023-07-20T17:48:39.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff76f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff7700ff720 0x7ff770068610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:39.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff76f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff7700ff720 0x7ff770068610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36302/0 (socket says 172.21.15.49:36302) 2023-07-20T17:48:39.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff76f7fe700 1 -- 172.21.15.49:0/3919790041 learned_addr learned my addr 172.21.15.49:0/3919790041 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:39.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff76effd700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff770100910 0x7ff77019b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:39.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff76f7fe700 1 -- 172.21.15.49:0/3919790041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff770100910 msgr2=0x7ff77019b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:39.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff76f7fe700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff770100910 0x7ff77019b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:39.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff76f7fe700 1 -- 172.21.15.49:0/3919790041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7701012b0 msgr2=0x7ff77019d930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T17:48:39.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff76f7fe700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7701012b0 0x7ff77019d930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:39.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff76f7fe700 1 -- 172.21.15.49:0/3919790041 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff768009d00 con 0x7ff7700ff720 2023-07-20T17:48:39.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.319+0000 7ff76f7fe700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff7700ff720 0x7ff770068610 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff76000d980 tx=0x7ff76000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:39.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.320+0000 7ff76cff9700 1 -- 172.21.15.49:0/3919790041 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760004020 con 0x7ff7700ff720 2023-07-20T17:48:39.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.320+0000 7ff775ec0700 1 -- 172.21.15.49:0/3919790041 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff770195680 con 0x7ff7700ff720 2023-07-20T17:48:39.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.320+0000 7ff76cff9700 1 -- 172.21.15.49:0/3919790041 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff760009730 con 0x7ff7700ff720 2023-07-20T17:48:39.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.320+0000 7ff76cff9700 1 -- 172.21.15.49:0/3919790041 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760003a30 con 0x7ff7700ff720 2023-07-20T17:48:39.327 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.320+0000 7ff775ec0700 1 -- 172.21.15.49:0/3919790041 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff770195be0 con 0x7ff7700ff720 2023-07-20T17:48:39.327 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.321+0000 7ff775ec0700 1 -- 172.21.15.49:0/3919790041 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77010af30 con 0x7ff7700ff720 2023-07-20T17:48:39.327 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.322+0000 7ff76cff9700 1 -- 172.21.15.49:0/3919790041 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff7600098a0 con 0x7ff7700ff720 2023-07-20T17:48:39.328 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.323+0000 7ff76cff9700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff7580a1150 0x7ff7580a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:39.328 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.323+0000 7ff76effd700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff7580a1150 0x7ff7580a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:39.328 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.323+0000 7ff76cff9700 1 -- 172.21.15.49:0/3919790041 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff7600cc4e0 con 0x7ff7700ff720 2023-07-20T17:48:39.328 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.324+0000 7ff76effd700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff7580a1150 0x7ff7580a3610 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff764009c60 tx=0x7ff764009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:39.330 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.327+0000 7ff76cff9700 1 -- 172.21.15.49:0/3919790041 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff7600960d0 con 0x7ff7700ff720 2023-07-20T17:48:39.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.583+0000 7ff775ec0700 1 -- 172.21.15.49:0/3919790041 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7701963c0 con 0x7ff7700ff720 2023-07-20T17:48:39.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.584+0000 7ff76cff9700 1 -- 172.21.15.49:0/3919790041 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7312 (secure 0 0 0) 0x7ff7600996f0 con 0x7ff7700ff720 2023-07-20T17:48:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:39.588 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1572141770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1572141770},{"type":"v1","addr":"172.21.15.49:6835","nonce":1572141770}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/4279110827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":4279110827},{"type":"v1","addr":"172.21.15.49:6837","nonce":4279110827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/1293946253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1293946253},{"type":"v1","addr":"172.21.15.97:6833","nonce":1293946253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2111036634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2111036634},{"type":"v1","addr":"172.21.15.79:6833","nonce":2111036634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/3824893609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":3824893609},{"type":"v1","addr":"172.21.15.79:6835","nonce":3824893609}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24484,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1698011127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1698011127},{"type":"v1","addr":"172.21.15.97:6835","nonce":1698011127}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/752441359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":752441359},{"type":"v1","addr":"172.21.15.97:6837","nonce":752441359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3288313742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3288313742},{"type":"v1","addr":"172.21.15.79:6837","nonce":3288313742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2639990675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2639990675},{"type":"v1","addr":"172.21.15.97:6839","nonce":2639990675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T17:48:39.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 -- 172.21.15.49:0/3919790041 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff7580a1150 msgr2=0x7ff7580a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:39.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff7580a1150 0x7ff7580a3610 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff764009c60 tx=0x7ff764009260 comp rx=0 tx=0).stop 2023-07-20T17:48:39.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 -- 172.21.15.49:0/3919790041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff7700ff720 msgr2=0x7ff770068610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:39.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff7700ff720 0x7ff770068610 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff76000d980 tx=0x7ff76000dc90 comp rx=0 tx=0).stop 2023-07-20T17:48:39.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 -- 172.21.15.49:0/3919790041 shutdown_connections 2023-07-20T17:48:39.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7ff7580a1150 0x7ff7580a3610 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:39.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7701012b0 0x7ff77019d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:39.591 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff7700ff720 0x7ff770068610 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:39.591 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 --2- 172.21.15.49:0/3919790041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff770100910 0x7ff77019b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:39.591 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 -- 172.21.15.49:0/3919790041 >> 172.21.15.49:0/3919790041 conn(0x7ff7700facb0 msgr2=0x7ff7700fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:39.591 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 -- 172.21.15.49:0/3919790041 shutdown_connections 2023-07-20T17:48:39.591 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:39.587+0000 7ff775ec0700 1 -- 172.21.15.49:0/3919790041 wait complete. 2023-07-20T17:48:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 10 2023-07-20T17:48:39.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:39 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:39.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:39 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:39.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:39 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.97:6838/2639990675,v1:172.21.15.97:6839/2639990675] up:boot 2023-07-20T17:48:39.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:39 smithi049 ceph-mon[96530]: fsmap 9 up:standby 2023-07-20T17:48:39.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:39 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:39.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:39 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T17:48:39.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:39 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:39.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:39 smithi049 ceph-mon[96530]: Deploying daemon mds.g on smithi049 2023-07-20T17:48:39.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:39 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3919790041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:40.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:39 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:40.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:39 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:40.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:39 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.97:6838/2639990675,v1:172.21.15.97:6839/2639990675] up:boot 2023-07-20T17:48:40.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:39 smithi097 ceph-mon[98398]: fsmap 9 up:standby 2023-07-20T17:48:40.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:39 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:40.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:39 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T17:48:40.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:39 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:40.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:39 smithi097 ceph-mon[98398]: Deploying daemon mds.g on smithi049 2023-07-20T17:48:40.019 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:39 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3919790041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:40.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:39 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:40.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:39 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:40.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:39 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.97:6838/2639990675,v1:172.21.15.97:6839/2639990675] up:boot 2023-07-20T17:48:40.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:39 smithi079 ceph-mon[98036]: fsmap 9 up:standby 2023-07-20T17:48:40.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:39 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:40.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:39 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T17:48:40.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:39 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:40.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:39 smithi079 ceph-mon[98036]: Deploying daemon mds.g on smithi049 2023-07-20T17:48:40.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:39 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3919790041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:40.491 INFO:journalctl@ceph.mds.g.smithi049.stdout:Jul 20 17:48:40 smithi049 systemd[1]: Starting Ceph mds.g for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:40.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:40 smithi049 ceph-mon[96530]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:40.937 INFO:journalctl@ceph.mds.g.smithi049.stdout:Jul 20 17:48:40 smithi049 podman[130981]: 2023-07-20T17:48:40.938 INFO:journalctl@ceph.mds.g.smithi049.stdout:Jul 20 17:48:40 smithi049 ceph-mds[131021]: -- 172.21.15.49:0/3069090043 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c56a5e51e0 con 0x55c56a5ba800 2023-07-20T17:48:40.938 INFO:journalctl@ceph.mds.g.smithi049.stdout:Jul 20 17:48:40 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-g[130997]: starting mds.g at 2023-07-20T17:48:41.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:40 smithi097 ceph-mon[98398]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:41.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:40 smithi079 ceph-mon[98036]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:41.240 INFO:journalctl@ceph.mds.g.smithi049.stdout:Jul 20 17:48:40 smithi049 bash[130981]: 2a049f843e8a7c852981049b242e4ba2c7a1e689d6b9fd8d3013775f4b78e26e 2023-07-20T17:48:41.240 INFO:journalctl@ceph.mds.g.smithi049.stdout:Jul 20 17:48:40 smithi049 systemd[1]: Started Ceph mds.g for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:41.605 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:41.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.739+0000 7faf08e20700 1 -- 172.21.15.49:0/1705220502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf04102520 msgr2=0x7faf04108dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:41.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.739+0000 7faf08e20700 1 --2- 172.21.15.49:0/1705220502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf04102520 0x7faf04108dd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faefc00b0a0 tx=0x7faefc00b3b0 comp rx=0 tx=0).stop 2023-07-20T17:48:41.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.739+0000 7faf08e20700 1 -- 172.21.15.49:0/1705220502 shutdown_connections 2023-07-20T17:48:41.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.739+0000 7faf08e20700 1 --2- 172.21.15.49:0/1705220502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf040feb10 0x7faf040fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:41.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.739+0000 7faf08e20700 1 --2- 172.21.15.49:0/1705220502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf04102520 0x7faf04108dd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:41.744 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.739+0000 7faf08e20700 1 --2- 172.21.15.49:0/1705220502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf04101b80 0x7faf04101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:41.744 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.739+0000 7faf08e20700 1 -- 172.21.15.49:0/1705220502 >> 172.21.15.49:0/1705220502 conn(0x7faf040fa420 msgr2=0x7faf040fc880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:41.744 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.739+0000 7faf08e20700 1 -- 172.21.15.49:0/1705220502 shutdown_connections 2023-07-20T17:48:41.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.739+0000 7faf08e20700 1 -- 172.21.15.49:0/1705220502 wait complete. 2023-07-20T17:48:41.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf08e20700 1 Processor -- start 2023-07-20T17:48:41.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf08e20700 1 -- start start 2023-07-20T17:48:41.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf08e20700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf040feb10 0x7faf04196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:41.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf08e20700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf04101b80 0x7faf04196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:41.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf08e20700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf04102520 0x7faf0419dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:41.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf08e20700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0419e1a0 con 0x7faf04102520 2023-07-20T17:48:41.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf08e20700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0419e2e0 con 0x7faf040feb10 2023-07-20T17:48:41.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf08e20700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0419e470 con 0x7faf04101b80 2023-07-20T17:48:41.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf02d9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf04102520 0x7faf0419dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:41.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf02d9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf04102520 0x7faf0419dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36328/0 (socket says 172.21.15.49:36328) 2023-07-20T17:48:41.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf02d9d700 1 -- 172.21.15.49:0/3302935178 learned_addr learned my addr 172.21.15.49:0/3302935178 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:41.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf01d9b700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf04101b80 0x7faf04196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:41.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf0259c700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf040feb10 0x7faf04196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:41.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf02d9d700 1 -- 172.21.15.49:0/3302935178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf04101b80 msgr2=0x7faf04196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:41.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf02d9d700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf04101b80 0x7faf04196b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:41.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf02d9d700 1 -- 172.21.15.49:0/3302935178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf040feb10 msgr2=0x7faf04196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:41.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.740+0000 7faf02d9d700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf040feb10 0x7faf04196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:41.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.741+0000 7faf02d9d700 1 -- 172.21.15.49:0/3302935178 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc009d00 con 0x7faf04102520 2023-07-20T17:48:41.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.741+0000 7faf0259c700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf040feb10 0x7faf04196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:48:41.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.741+0000 7faf02d9d700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf04102520 0x7faf0419dbd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faefc00b070 tx=0x7faefc004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:41.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.741+0000 7faef37fe700 1 -- 172.21.15.49:0/3302935178 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faefc00e040 con 0x7faf04102520 2023-07-20T17:48:41.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.741+0000 7faf08e20700 1 -- 172.21.15.49:0/3302935178 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0419e6f0 con 0x7faf04102520 2023-07-20T17:48:41.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.741+0000 7faef37fe700 1 -- 172.21.15.49:0/3302935178 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faefc004e80 con 0x7faf04102520 2023-07-20T17:48:41.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.741+0000 7faef37fe700 1 -- 172.21.15.49:0/3302935178 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faefc01b6b0 con 0x7faf04102520 2023-07-20T17:48:41.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.741+0000 7faf08e20700 1 -- 172.21.15.49:0/3302935178 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0419eb20 con 0x7faf04102520 2023-07-20T17:48:41.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.743+0000 7faf08e20700 1 -- 172.21.15.49:0/3302935178 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf04107e80 con 0x7faf04102520 2023-07-20T17:48:41.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.743+0000 7faef37fe700 1 -- 172.21.15.49:0/3302935178 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faefc003a00 con 0x7faf04102520 2023-07-20T17:48:41.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.744+0000 7faef37fe700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7faeec0a1150 0x7faeec0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:41.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.744+0000 7faef37fe700 1 -- 172.21.15.49:0/3302935178 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7faefc0cd9d0 con 0x7faf04102520 2023-07-20T17:48:41.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.744+0000 7faf0259c700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7faeec0a1150 0x7faeec0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:41.753 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.744+0000 7faf0259c700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7faeec0a1150 0x7faeec0a3610 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faef4005950 tx=0x7faef40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:41.754 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:41.749+0000 7faef37fe700 1 -- 172.21.15.49:0/3302935178 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faefc097650 con 0x7faf04102520 2023-07-20T17:48:42.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.001+0000 7faf08e20700 1 -- 172.21.15.49:0/3302935178 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf0402cb20 con 0x7faf04102520 2023-07-20T17:48:42.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.002+0000 7faef37fe700 1 -- 172.21.15.49:0/3302935178 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8067 (secure 0 0 0) 0x7faefc09ac70 con 0x7faf04102520 2023-07-20T17:48:42.005 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:42.005 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1572141770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1572141770},{"type":"v1","addr":"172.21.15.49:6835","nonce":1572141770}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/4279110827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":4279110827},{"type":"v1","addr":"172.21.15.49:6837","nonce":4279110827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6839/591358452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":591358452},{"type":"v1","addr":"172.21.15.49:6839","nonce":591358452}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/1293946253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1293946253},{"type":"v1","addr":"172.21.15.97:6833","nonce":1293946253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2111036634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2111036634},{"type":"v1","addr":"172.21.15.79:6833","nonce":2111036634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/3824893609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":3824893609},{"type":"v1","addr":"172.21.15.79:6835","nonce":3824893609}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24484,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1698011127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1698011127},{"type":"v1","addr":"172.21.15.97:6835","nonce":1698011127}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/752441359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":752441359},{"type":"v1","addr":"172.21.15.97:6837","nonce":752441359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3288313742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3288313742},{"type":"v1","addr":"172.21.15.79:6837","nonce":3288313742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2639990675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2639990675},{"type":"v1","addr":"172.21.15.97:6839","nonce":2639990675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T17:48:42.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 -- 172.21.15.49:0/3302935178 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7faeec0a1150 msgr2=0x7faeec0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:42.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7faeec0a1150 0x7faeec0a3610 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faef4005950 tx=0x7faef40058e0 comp rx=0 tx=0).stop 2023-07-20T17:48:42.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 -- 172.21.15.49:0/3302935178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf04102520 msgr2=0x7faf0419dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:42.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf04102520 0x7faf0419dbd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faefc00b070 tx=0x7faefc004a50 comp rx=0 tx=0).stop 2023-07-20T17:48:42.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 -- 172.21.15.49:0/3302935178 shutdown_connections 2023-07-20T17:48:42.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7faeec0a1150 0x7faeec0a3610 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:42.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf04102520 0x7faf0419dbd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:42.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf040feb10 0x7faf04196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:42.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 --2- 172.21.15.49:0/3302935178 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf04101b80 0x7faf04196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:42.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 -- 172.21.15.49:0/3302935178 >> 172.21.15.49:0/3302935178 conn(0x7faf040fa420 msgr2=0x7faf04104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:42.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 -- 172.21.15.49:0/3302935178 shutdown_connections 2023-07-20T17:48:42.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:42.005+0000 7faf08e20700 1 -- 172.21.15.49:0/3302935178 wait complete. 2023-07-20T17:48:42.009 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 11 2023-07-20T17:48:42.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:42.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: Deploying daemon mds.a on smithi049 2023-07-20T17:48:42.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.49:6838/591358452,v1:172.21.15.49:6839/591358452] up:boot 2023-07-20T17:48:42.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: fsmap 10 up:standby 2023-07-20T17:48:42.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T17:48:42.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:42 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3302935178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:42.317 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:42.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:42.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:42.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:42.318 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: Deploying daemon mds.a on smithi049 2023-07-20T17:48:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.49:6838/591358452,v1:172.21.15.49:6839/591358452] up:boot 2023-07-20T17:48:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: fsmap 10 up:standby 2023-07-20T17:48:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T17:48:42.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:42 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3302935178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:42.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:42.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:42.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:42.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:42.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:42.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:42.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:42.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: Deploying daemon mds.a on smithi049 2023-07-20T17:48:42.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.49:6838/591358452,v1:172.21.15.49:6839/591358452] up:boot 2023-07-20T17:48:42.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: fsmap 10 up:standby 2023-07-20T17:48:42.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T17:48:42.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:42 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3302935178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:42.982 INFO:journalctl@ceph.mds.a.smithi049.stdout:Jul 20 17:48:42 smithi049 systemd[1]: Starting Ceph mds.a for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:42.983 INFO:journalctl@ceph.mds.a.smithi049.stdout:Jul 20 17:48:42 smithi049 podman[131364]: 2023-07-20T17:48:43.740 INFO:journalctl@ceph.mds.a.smithi049.stdout:Jul 20 17:48:43 smithi049 ceph-mds[131404]: -- 172.21.15.49:0/468398227 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e85c1f11e0 con 0x55e85c1c6800 2023-07-20T17:48:43.741 INFO:journalctl@ceph.mds.a.smithi049.stdout:Jul 20 17:48:43 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-a[131380]: starting mds.a at 2023-07-20T17:48:43.741 INFO:journalctl@ceph.mds.a.smithi049.stdout:Jul 20 17:48:43 smithi049 bash[131364]: 157b93612343a352e58bdf374971bdd4ad91b9b728cfb966fbce906e5908aecf 2023-07-20T17:48:43.741 INFO:journalctl@ceph.mds.a.smithi049.stdout:Jul 20 17:48:43 smithi049 systemd[1]: Started Ceph mds.a for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:44.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:44.166 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.163+0000 7f6cea899700 1 -- 172.21.15.49:0/332443745 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce41012b0 msgr2=0x7f6ce4107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:44.166 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.163+0000 7f6cea899700 1 --2- 172.21.15.49:0/332443745 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce41012b0 0x7f6ce4107b60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00b0d0 tx=0x7f6cdc00b3e0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.166 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.163+0000 7f6cea899700 1 -- 172.21.15.49:0/332443745 shutdown_connections 2023-07-20T17:48:44.166 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.163+0000 7f6cea899700 1 --2- 172.21.15.49:0/332443745 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce41012b0 0x7f6ce4107b60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.166 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.163+0000 7f6cea899700 1 --2- 172.21.15.49:0/332443745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ce40ff720 0x7f6ce40ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.167 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.163+0000 7f6cea899700 1 --2- 172.21.15.49:0/332443745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ce4100910 0x7f6ce4100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.167 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.163+0000 7f6cea899700 1 -- 172.21.15.49:0/332443745 >> 172.21.15.49:0/332443745 conn(0x7f6ce40facb0 msgr2=0x7f6ce40fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:44.167 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.163+0000 7f6cea899700 1 -- 172.21.15.49:0/332443745 shutdown_connections 2023-07-20T17:48:44.167 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.163+0000 7f6cea899700 1 -- 172.21.15.49:0/332443745 wait complete. 2023-07-20T17:48:44.167 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.163+0000 7f6cea899700 1 Processor -- start 2023-07-20T17:48:44.167 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.163+0000 7f6cea899700 1 -- start start 2023-07-20T17:48:44.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6cea899700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ce40ff720 0x7f6ce41954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:44.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6cea899700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ce4100910 0x7f6ce41959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:44.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6cea899700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce41012b0 0x7f6ce419ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:44.171 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6cea899700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce419d030 con 0x7f6ce41012b0 2023-07-20T17:48:44.171 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6cea899700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce419d170 con 0x7f6ce4100910 2023-07-20T17:48:44.171 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6cea899700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce419d300 con 0x7f6ce40ff720 2023-07-20T17:48:44.171 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce8e36700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce41012b0 0x7f6ce419ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:44.171 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce8e36700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce41012b0 0x7f6ce419ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36356/0 (socket says 172.21.15.49:36356) 2023-07-20T17:48:44.171 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce8e36700 1 -- 172.21.15.49:0/2640501399 learned_addr learned my addr 172.21.15.49:0/2640501399 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:44.171 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6cdbfff700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ce4100910 0x7f6ce41959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:44.171 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce3fff700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ce40ff720 0x7f6ce41954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:44.172 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce8e36700 1 -- 172.21.15.49:0/2640501399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ce40ff720 msgr2=0x7f6ce41954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:44.172 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce8e36700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ce40ff720 0x7f6ce41954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.172 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce8e36700 1 -- 172.21.15.49:0/2640501399 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ce4100910 msgr2=0x7f6ce41959f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:44.172 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce8e36700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ce4100910 0x7f6ce41959f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.172 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce8e36700 1 -- 172.21.15.49:0/2640501399 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc009d00 con 0x7f6ce41012b0 2023-07-20T17:48:44.172 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce8e36700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce41012b0 0x7f6ce419ca60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc007a00 tx=0x7f6cdc0094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:44.172 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce1ffb700 1 -- 172.21.15.49:0/2640501399 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc00e040 con 0x7f6ce41012b0 2023-07-20T17:48:44.173 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce1ffb700 1 -- 172.21.15.49:0/2640501399 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6cdc003e10 con 0x7f6ce41012b0 2023-07-20T17:48:44.173 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6cea899700 1 -- 172.21.15.49:0/2640501399 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce419d580 con 0x7f6ce41012b0 2023-07-20T17:48:44.173 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6ce1ffb700 1 -- 172.21.15.49:0/2640501399 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc012a90 con 0x7f6ce41012b0 2023-07-20T17:48:44.173 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.164+0000 7f6cea899700 1 -- 172.21.15.49:0/2640501399 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce419d9a0 con 0x7f6ce41012b0 2023-07-20T17:48:44.173 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.165+0000 7f6cea899700 1 -- 172.21.15.49:0/2640501399 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce410af30 con 0x7f6ce41012b0 2023-07-20T17:48:44.173 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.166+0000 7f6ce1ffb700 1 -- 172.21.15.49:0/2640501399 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6cdc0056c0 con 0x7f6ce41012b0 2023-07-20T17:48:44.173 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.167+0000 7f6ce1ffb700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6cd00a1150 0x7f6cd00a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:44.174 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.167+0000 7f6ce1ffb700 1 -- 172.21.15.49:0/2640501399 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6cdc0ce2d0 con 0x7f6ce41012b0 2023-07-20T17:48:44.174 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.167+0000 7f6ce3fff700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6cd00a1150 0x7f6cd00a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:44.174 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.168+0000 7f6ce3fff700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6cd00a1150 0x7f6cd00a3610 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4005950 tx=0x7f6cd40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:44.174 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.171+0000 7f6ce1ffb700 1 -- 172.21.15.49:0/2640501399 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6cdc097f50 con 0x7f6ce41012b0 2023-07-20T17:48:44.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.419+0000 7f6cea899700 1 -- 172.21.15.49:0/2640501399 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ce41065d0 con 0x7f6ce41012b0 2023-07-20T17:48:44.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.419+0000 7f6ce1ffb700 1 -- 172.21.15.49:0/2640501399 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8825 (secure 0 0 0) 0x7f6cdc09b570 con 0x7f6ce41012b0 2023-07-20T17:48:44.422 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:44.422 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1572141770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1572141770},{"type":"v1","addr":"172.21.15.49:6835","nonce":1572141770}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/4279110827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":4279110827},{"type":"v1","addr":"172.21.15.49:6837","nonce":4279110827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6839/591358452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":591358452},{"type":"v1","addr":"172.21.15.49:6839","nonce":591358452}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6841/3542151457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3542151457},{"type":"v1","addr":"172.21.15.49:6841","nonce":3542151457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/1293946253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1293946253},{"type":"v1","addr":"172.21.15.97:6833","nonce":1293946253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2111036634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2111036634},{"type":"v1","addr":"172.21.15.79:6833","nonce":2111036634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/3824893609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":3824893609},{"type":"v1","addr":"172.21.15.79:6835","nonce":3824893609}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24484,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1698011127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1698011127},{"type":"v1","addr":"172.21.15.97:6835","nonce":1698011127}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/752441359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":752441359},{"type":"v1","addr":"172.21.15.97:6837","nonce":752441359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3288313742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3288313742},{"type":"v1","addr":"172.21.15.79:6837","nonce":3288313742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2639990675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2639990675},{"type":"v1","addr":"172.21.15.97:6839","nonce":2639990675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T17:48:44.424 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.421+0000 7f6cea899700 1 -- 172.21.15.49:0/2640501399 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6cd00a1150 msgr2=0x7f6cd00a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:44.424 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.421+0000 7f6cea899700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6cd00a1150 0x7f6cd00a3610 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4005950 tx=0x7f6cd40058e0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.421+0000 7f6cea899700 1 -- 172.21.15.49:0/2640501399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce41012b0 msgr2=0x7f6ce419ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:44.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.421+0000 7f6cea899700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce41012b0 0x7f6ce419ca60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc007a00 tx=0x7f6cdc0094c0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.422+0000 7f6cea899700 1 -- 172.21.15.49:0/2640501399 shutdown_connections 2023-07-20T17:48:44.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.422+0000 7f6cea899700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6cd00a1150 0x7f6cd00a3610 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.422+0000 7f6cea899700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6ce40ff720 0x7f6ce41954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.422+0000 7f6cea899700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce41012b0 0x7f6ce419ca60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.422+0000 7f6cea899700 1 --2- 172.21.15.49:0/2640501399 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ce4100910 0x7f6ce41959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:44.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.422+0000 7f6cea899700 1 -- 172.21.15.49:0/2640501399 >> 172.21.15.49:0/2640501399 conn(0x7f6ce40facb0 msgr2=0x7f6ce40fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:44.426 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.422+0000 7f6cea899700 1 -- 172.21.15.49:0/2640501399 shutdown_connections 2023-07-20T17:48:44.426 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:44.422+0000 7f6cea899700 1 -- 172.21.15.49:0/2640501399 wait complete. 2023-07-20T17:48:44.426 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 12 2023-07-20T17:48:44.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:44 smithi079 ceph-mon[98036]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:44.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:44.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:44.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:44 smithi079 ceph-mon[98036]: Deploying daemon mds.b on smithi079 2023-07-20T17:48:44.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:44 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.49:6840/3542151457,v1:172.21.15.49:6841/3542151457] up:boot 2023-07-20T17:48:44.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:44 smithi079 ceph-mon[98036]: fsmap 11 up:standby 2023-07-20T17:48:44.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:44 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T17:48:44.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:44 smithi049 ceph-mon[96530]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:44.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:44.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:44 smithi049 ceph-mon[96530]: Deploying daemon mds.b on smithi079 2023-07-20T17:48:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:44 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.49:6840/3542151457,v1:172.21.15.49:6841/3542151457] up:boot 2023-07-20T17:48:44.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:44 smithi049 ceph-mon[96530]: fsmap 11 up:standby 2023-07-20T17:48:44.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:44 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T17:48:44.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:44 smithi097 ceph-mon[98398]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:44.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:44.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:44.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:44.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T17:48:44.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T17:48:44.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:44.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:44 smithi097 ceph-mon[98398]: Deploying daemon mds.b on smithi079 2023-07-20T17:48:44.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:44 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.49:6840/3542151457,v1:172.21.15.49:6841/3542151457] up:boot 2023-07-20T17:48:44.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:44 smithi097 ceph-mon[98398]: fsmap 11 up:standby 2023-07-20T17:48:44.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:44 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T17:48:45.068 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:48:44 smithi079 systemd[1]: Starting Ceph mds.b for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:45.387 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:45 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/2640501399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:45.388 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:48:45 smithi079 podman[123316]: 2023-07-20T17:48:45.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:45 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/2640501399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:45.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:45 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/2640501399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:45.817 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:48:45 smithi079 bash[123316]: 87eba083bccda2fc27c667908ddee89705ffa9f87a9a085a15150de7977d448b 2023-07-20T17:48:45.818 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:48:45 smithi079 ceph-mds[123356]: -- 172.21.15.79:0/776967942 <== mon.2 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564ebddf71e0 con 0x564ebddcc800 2023-07-20T17:48:45.818 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:48:45 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-b[123332]: starting mds.b at 2023-07-20T17:48:45.818 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:48:45 smithi079 systemd[1]: Started Ceph mds.b for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:48:46.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T17:48:46.573 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.565+0000 7f324f4f4700 1 -- 172.21.15.49:0/34078261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3248100830 msgr2=0x7f3248100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:46.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.565+0000 7f324f4f4700 1 --2- 172.21.15.49:0/34078261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3248100830 0x7f3248100c50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f323c0099c0 tx=0x7f323c009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.565+0000 7f324f4f4700 1 -- 172.21.15.49:0/34078261 shutdown_connections 2023-07-20T17:48:46.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.565+0000 7f324f4f4700 1 --2- 172.21.15.49:0/34078261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32481023c0 0x7f3248108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.565+0000 7f324f4f4700 1 --2- 172.21.15.49:0/34078261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3248100830 0x7f3248100c50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.565+0000 7f324f4f4700 1 --2- 172.21.15.49:0/34078261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248101a20 0x7f3248101e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.565+0000 7f324f4f4700 1 -- 172.21.15.49:0/34078261 >> 172.21.15.49:0/34078261 conn(0x7f32480fbdc0 msgr2=0x7f32480fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:46.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.565+0000 7f324f4f4700 1 -- 172.21.15.49:0/34078261 shutdown_connections 2023-07-20T17:48:46.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324f4f4700 1 -- 172.21.15.49:0/34078261 wait complete. 2023-07-20T17:48:46.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324f4f4700 1 Processor -- start 2023-07-20T17:48:46.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324f4f4700 1 -- start start 2023-07-20T17:48:46.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324f4f4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3248100830 0x7f3248196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:46.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324f4f4700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248101a20 0x7f3248196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:46.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324f4f4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32481023c0 0x7f324819db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:46.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324f4f4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324819e0e0 con 0x7f32481023c0 2023-07-20T17:48:46.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324f4f4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324819e220 con 0x7f3248100830 2023-07-20T17:48:46.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324f4f4700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324819e3b0 con 0x7f3248101a20 2023-07-20T17:48:46.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324da91700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32481023c0 0x7f324819db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:46.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324d290700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3248100830 0x7f3248196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:46.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324da91700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32481023c0 0x7f324819db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36370/0 (socket says 172.21.15.49:36370) 2023-07-20T17:48:46.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324da91700 1 -- 172.21.15.49:0/1346786036 learned_addr learned my addr 172.21.15.49:0/1346786036 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:46.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.566+0000 7f324ca8f700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248101a20 0x7f3248196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:46.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.567+0000 7f324da91700 1 -- 172.21.15.49:0/1346786036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248101a20 msgr2=0x7f3248196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:46.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.567+0000 7f324da91700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248101a20 0x7f3248196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.567+0000 7f324da91700 1 -- 172.21.15.49:0/1346786036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3248100830 msgr2=0x7f3248196560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:46.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.567+0000 7f324da91700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3248100830 0x7f3248196560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.567+0000 7f324da91700 1 -- 172.21.15.49:0/1346786036 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c0095f0 con 0x7f32481023c0 2023-07-20T17:48:46.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.567+0000 7f324da91700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32481023c0 0x7f324819db10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3240007ee0 tx=0x7f324000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:46.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.567+0000 7f323a7fc700 1 -- 172.21.15.49:0/1346786036 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324000dd20 con 0x7f32481023c0 2023-07-20T17:48:46.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.567+0000 7f323a7fc700 1 -- 172.21.15.49:0/1346786036 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f324000f040 con 0x7f32481023c0 2023-07-20T17:48:46.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.567+0000 7f324f4f4700 1 -- 172.21.15.49:0/1346786036 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324819e690 con 0x7f32481023c0 2023-07-20T17:48:46.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.567+0000 7f323a7fc700 1 -- 172.21.15.49:0/1346786036 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32400048a0 con 0x7f32481023c0 2023-07-20T17:48:46.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.567+0000 7f324f4f4700 1 -- 172.21.15.49:0/1346786036 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324819eb50 con 0x7f32481023c0 2023-07-20T17:48:46.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.568+0000 7f324f4f4700 1 -- 172.21.15.49:0/1346786036 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3248107e20 con 0x7f32481023c0 2023-07-20T17:48:46.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.569+0000 7f323a7fc700 1 -- 172.21.15.49:0/1346786036 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3240003a00 con 0x7f32481023c0 2023-07-20T17:48:46.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.570+0000 7f323a7fc700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f32340a1150 0x7f32340a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:46.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.570+0000 7f324d290700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f32340a1150 0x7f32340a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:46.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.570+0000 7f323a7fc700 1 -- 172.21.15.49:0/1346786036 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f32400cbcd0 con 0x7f32481023c0 2023-07-20T17:48:46.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.570+0000 7f324d290700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f32340a1150 0x7f32340a3610 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f323c00b580 tx=0x7f323c009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:46.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.574+0000 7f323a7fc700 1 -- 172.21.15.49:0/1346786036 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f324004e780 con 0x7f32481023c0 2023-07-20T17:48:46.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:46 smithi049 ceph-mon[96530]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:46.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:46 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.79:6838/2994701526,v1:172.21.15.79:6839/2994701526] up:boot 2023-07-20T17:48:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:46 smithi049 ceph-mon[96530]: fsmap 12 up:standby 2023-07-20T17:48:46.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:46 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T17:48:46.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:46 smithi097 ceph-mon[98398]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:46.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:46.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:46 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.79:6838/2994701526,v1:172.21.15.79:6839/2994701526] up:boot 2023-07-20T17:48:46.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:46 smithi097 ceph-mon[98398]: fsmap 12 up:standby 2023-07-20T17:48:46.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:46 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T17:48:46.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:46 smithi079 ceph-mon[98036]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:46.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:46 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.79:6838/2994701526,v1:172.21.15.79:6839/2994701526] up:boot 2023-07-20T17:48:46.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:46 smithi079 ceph-mon[98036]: fsmap 12 up:standby 2023-07-20T17:48:46.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:46 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T17:48:46.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.838+0000 7f324f4f4700 1 -- 172.21.15.49:0/1346786036 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f324802c970 con 0x7f32481023c0 2023-07-20T17:48:46.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.838+0000 7f323a7fc700 1 -- 172.21.15.49:0/1346786036 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9583 (secure 0 0 0) 0x7f32400958c0 con 0x7f32481023c0 2023-07-20T17:48:46.841 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:46.841 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1572141770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1572141770},{"type":"v1","addr":"172.21.15.49:6835","nonce":1572141770}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/4279110827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":4279110827},{"type":"v1","addr":"172.21.15.49:6837","nonce":4279110827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6839/591358452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":591358452},{"type":"v1","addr":"172.21.15.49:6839","nonce":591358452}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6841/3542151457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3542151457},{"type":"v1","addr":"172.21.15.49:6841","nonce":3542151457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6833/1293946253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1293946253},{"type":"v1","addr":"172.21.15.97:6833","nonce":1293946253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6833/2111036634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2111036634},{"type":"v1","addr":"172.21.15.79:6833","nonce":2111036634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/3824893609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":3824893609},{"type":"v1","addr":"172.21.15.79:6835","nonce":3824893609}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24484,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/1698011127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":1698011127},{"type":"v1","addr":"172.21.15.97:6835","nonce":1698011127}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/752441359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":752441359},{"type":"v1","addr":"172.21.15.97:6837","nonce":752441359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/3288313742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":3288313742},{"type":"v1","addr":"172.21.15.79:6837","nonce":3288313742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2639990675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2639990675},{"type":"v1","addr":"172.21.15.97:6839","nonce":2639990675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/2994701526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":2994701526},{"type":"v1","addr":"172.21.15.79:6839","nonce":2994701526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-20T17:48:46.843 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 -- 172.21.15.49:0/1346786036 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f32340a1150 msgr2=0x7f32340a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:46.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f32340a1150 0x7f32340a3610 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f323c00b580 tx=0x7f323c009f10 comp rx=0 tx=0).stop 2023-07-20T17:48:46.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 -- 172.21.15.49:0/1346786036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32481023c0 msgr2=0x7f324819db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:46.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32481023c0 0x7f324819db10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3240007ee0 tx=0x7f324000d3b0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 -- 172.21.15.49:0/1346786036 shutdown_connections 2023-07-20T17:48:46.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f32340a1150 0x7f32340a3610 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32481023c0 0x7f324819db10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3248100830 0x7f3248196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 --2- 172.21.15.49:0/1346786036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3248101a20 0x7f3248196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 -- 172.21.15.49:0/1346786036 >> 172.21.15.49:0/1346786036 conn(0x7f32480fbdc0 msgr2=0x7f3248104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:46.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 -- 172.21.15.49:0/1346786036 shutdown_connections 2023-07-20T17:48:46.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.841+0000 7f324f4f4700 1 -- 172.21.15.49:0/1346786036 wait complete. 2023-07-20T17:48:46.845 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 13 2023-07-20T17:48:46.856 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T17:48:46.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.984+0000 7fa0d588f700 1 -- 172.21.15.49:0/1361932625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d00fee60 msgr2=0x7fa0d00ff280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:46.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.984+0000 7fa0d588f700 1 --2- 172.21.15.49:0/1361932625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d00fee60 0x7fa0d00ff280 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009990 tx=0x7fa0c0009ca0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.985+0000 7fa0d588f700 1 -- 172.21.15.49:0/1361932625 shutdown_connections 2023-07-20T17:48:46.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.985+0000 7fa0d588f700 1 --2- 172.21.15.49:0/1361932625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0d0100a00 0x7fa0d0107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.985+0000 7fa0d588f700 1 --2- 172.21.15.49:0/1361932625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d00fee60 0x7fa0d00ff280 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.985+0000 7fa0d588f700 1 --2- 172.21.15.49:0/1361932625 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d0100040 0x7fa0d01004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.985+0000 7fa0d588f700 1 -- 172.21.15.49:0/1361932625 >> 172.21.15.49:0/1361932625 conn(0x7fa0d00fa3e0 msgr2=0x7fa0d00fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:46.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.985+0000 7fa0d588f700 1 -- 172.21.15.49:0/1361932625 shutdown_connections 2023-07-20T17:48:46.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.985+0000 7fa0d588f700 1 -- 172.21.15.49:0/1361932625 wait complete. 2023-07-20T17:48:46.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.985+0000 7fa0d588f700 1 Processor -- start 2023-07-20T17:48:46.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.985+0000 7fa0d588f700 1 -- start start 2023-07-20T17:48:46.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0d588f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d00fee60 0x7fa0d0195440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:46.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0d588f700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d0100040 0x7fa0d0195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:46.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0d588f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0d0100a00 0x7fa0d019c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:46.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0d588f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d019cfc0 con 0x7fa0d00fee60 2023-07-20T17:48:46.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0d588f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d019d100 con 0x7fa0d0100a00 2023-07-20T17:48:46.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0d588f700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d019d290 con 0x7fa0d0100040 2023-07-20T17:48:46.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0ceffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d00fee60 0x7fa0d0195440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:46.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0ceffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d00fee60 0x7fa0d0195440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36380/0 (socket says 172.21.15.49:36380) 2023-07-20T17:48:46.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0ceffd700 1 -- 172.21.15.49:0/3552445621 learned_addr learned my addr 172.21.15.49:0/3552445621 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:46.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0ce7fc700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d0100040 0x7fa0d0195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:46.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0cf7fe700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0d0100a00 0x7fa0d019c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:46.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0ceffd700 1 -- 172.21.15.49:0/3552445621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d0100040 msgr2=0x7fa0d0195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:46.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0ceffd700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d0100040 0x7fa0d0195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0ceffd700 1 -- 172.21.15.49:0/3552445621 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0d0100a00 msgr2=0x7fa0d019c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:46.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0ceffd700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0d0100a00 0x7fa0d019c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:46.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0ceffd700 1 -- 172.21.15.49:0/3552445621 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c00095f0 con 0x7fa0d00fee60 2023-07-20T17:48:46.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0cf7fe700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0d0100a00 0x7fa0d019c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T17:48:46.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0ceffd700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d00fee60 0x7fa0d0195440 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00095c0 tx=0x7fa0c001a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:46.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0d488d700 1 -- 172.21.15.49:0/3552445621 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c001f070 con 0x7fa0d00fee60 2023-07-20T17:48:46.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0d488d700 1 -- 172.21.15.49:0/3552445621 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0c000f460 con 0x7fa0d00fee60 2023-07-20T17:48:46.993 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0d588f700 1 -- 172.21.15.49:0/3552445621 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d0107c50 con 0x7fa0d00fee60 2023-07-20T17:48:46.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0d488d700 1 -- 172.21.15.49:0/3552445621 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0025660 con 0x7fa0d00fee60 2023-07-20T17:48:46.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.986+0000 7fa0d588f700 1 -- 172.21.15.49:0/3552445621 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d0108150 con 0x7fa0d00fee60 2023-07-20T17:48:46.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.987+0000 7fa0d588f700 1 -- 172.21.15.49:0/3552445621 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d0108b40 con 0x7fa0d00fee60 2023-07-20T17:48:46.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.989+0000 7fa0d488d700 1 -- 172.21.15.49:0/3552445621 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa0c0023070 con 0x7fa0d00fee60 2023-07-20T17:48:46.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.990+0000 7fa0d488d700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa0b80a11d0 0x7fa0b80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:46.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.990+0000 7fa0d488d700 1 -- 172.21.15.49:0/3552445621 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa0c00cfc70 con 0x7fa0d00fee60 2023-07-20T17:48:46.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.990+0000 7fa0ce7fc700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa0b80a11d0 0x7fa0b80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:46.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.990+0000 7fa0ce7fc700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa0b80a11d0 0x7fa0b80a3690 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00feb80 tx=0x7fa0c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:46.996 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:46.993+0000 7fa0d488d700 1 -- 172.21.15.49:0/3552445621 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa0c00998f0 con 0x7fa0d00fee60 2023-07-20T17:48:47.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.207+0000 7fa0d588f700 1 -- 172.21.15.49:0/3552445621 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0d0105840 con 0x7fa0d00fee60 2023-07-20T17:48:47.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.209+0000 7fa0d488d700 1 -- 172.21.15.49:0/3552445621 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18314 (secure 0 0 0) 0x7fa0c009cf10 con 0x7fa0d00fee60 2023-07-20T17:48:47.212 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:47.212 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":78,"fsid":"8c050cea-2724-11ee-9b34-001a4aab830c","created":"2023-07-20T17:41:57.467323+0000","modified":"2023-07-20T17:48:10.226553+0000","last_up_change":"2023-07-20T17:48:09.220884+0000","last_in_change":"2023-07-20T17:47:51.156799+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T17:45:01.297180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63bfce41-a0a8-49b6-aa1e-ed7809c0627e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6803","nonce":2306317776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6805","nonce":2306317776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6809","nonce":2306317776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6807","nonce":2306317776}]},"public_addr":"172.21.15.49:6803/2306317776","cluster_addr":"172.21.15.49:6805/2306317776","heartbeat_back_addr":"172.21.15.49:6809/2306317776","heartbeat_front_addr":"172.21.15.49:6807/2306317776","state":["exists","up"]},{"osd":1,"uuid":"48cd9d65-b37d-410a-8007-f53fbd24dd73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6801","nonce":2048064928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6803","nonce":2048064928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6807","nonce":2048064928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6805","nonce":2048064928}]},"public_addr":"172.21.15.79:6801/2048064928","cluster_addr":"172.21.15.79:6803/2048064928","heartbeat_back_addr":"172.21.15.79:6807/2048064928","heartbeat_front_addr":"172.21.15.79:6805/2048064928","state":["exists","up"]},{"osd":2,"uuid":"272a17dd-e825-4990-a178-c2eb6985f37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6801","nonce":2945387274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6803","nonce":2945387274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6807","nonce":2945387274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6805","nonce":2945387274}]},"public_addr":"172.21.15.97:6801/2945387274","cluster_addr":"172.21.15.97:6803/2945387274","heartbeat_back_addr":"172.21.15.97:6807/2945387274","heartbeat_front_addr":"172.21.15.97:6805/2945387274","state":["exists","up"]},{"osd":3,"uuid":"6ba1972a-2bf4-4602-900c-11b15dcf6027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6811","nonce":178015359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6813","nonce":178015359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6817","nonce":178015359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6815","nonce":178015359}]},"public_addr":"172.21.15.49:6811/178015359","cluster_addr":"172.21.15.49:6813/178015359","heartbeat_back_addr":"172.21.15.49:6817/178015359","heartbeat_front_addr":"172.21.15.49:6815/178015359","state":["exists","up"]},{"osd":4,"uuid":"a167452b-f627-4aae-aecd-e513a5649235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6809","nonce":4037011752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6811","nonce":4037011752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6815","nonce":4037011752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6813","nonce":4037011752}]},"public_addr":"172.21.15.79:6809/4037011752","cluster_addr":"172.21.15.79:6811/4037011752","heartbeat_back_addr":"172.21.15.79:6815/4037011752","heartbeat_front_addr":"172.21.15.79:6813/4037011752","state":["exists","up"]},{"osd":5,"uuid":"def1f97c-84c4-4c87-9363-62f72142e0a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6809","nonce":2172761707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6811","nonce":2172761707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6815","nonce":2172761707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6813","nonce":2172761707}]},"public_addr":"172.21.15.97:6809/2172761707","cluster_addr":"172.21.15.97:6811/2172761707","heartbeat_back_addr":"172.21.15.97:6815/2172761707","heartbeat_front_addr":"172.21.15.97:6813/2172761707","state":["exists","up"]},{"osd":6,"uuid":"e8c200c2-0692-4d5e-a281-77c9ad6670cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6819","nonce":797242207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6821","nonce":797242207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6825","nonce":797242207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6823","nonce":797242207}]},"public_addr":"172.21.15.49:6819/797242207","cluster_addr":"172.21.15.49:6821/797242207","heartbeat_back_addr":"172.21.15.49:6825/797242207","heartbeat_front_addr":"172.21.15.49:6823/797242207","state":["exists","up"]},{"osd":7,"uuid":"847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6817","nonce":1417643292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6819","nonce":1417643292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6823","nonce":1417643292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6821","nonce":1417643292}]},"public_addr":"172.21.15.79:6817/1417643292","cluster_addr":"172.21.15.79:6819/1417643292","heartbeat_back_addr":"172.21.15.79:6823/1417643292","heartbeat_front_addr":"172.21.15.79:6821/1417643292","state":["exists","up"]},{"osd":8,"uuid":"14c55ac6-0e96-42a4-b974-178c87c2185c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6817","nonce":1465924719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6819","nonce":1465924719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6823","nonce":1465924719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6821","nonce":1465924719}]},"public_addr":"172.21.15.97:6817/1465924719","cluster_addr":"172.21.15.97:6819/1465924719","heartbeat_back_addr":"172.21.15.97:6823/1465924719","heartbeat_front_addr":"172.21.15.97:6821/1465924719","state":["exists","up"]},{"osd":9,"uuid":"b0c07e94-8039-4e41-85b7-23344124687b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6827","nonce":845176131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6829","nonce":845176131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6833","nonce":845176131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6831","nonce":845176131}]},"public_addr":"172.21.15.49:6827/845176131","cluster_addr":"172.21.15.49:6829/845176131","heartbeat_back_addr":"172.21.15.49:6833/845176131","heartbeat_front_addr":"172.21.15.49:6831/845176131","state":["exists","up"]},{"osd":10,"uuid":"67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6825","nonce":935518328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6827","nonce":935518328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6831","nonce":935518328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6829","nonce":935518328}]},"public_addr":"172.21.15.79:6825/935518328","cluster_addr":"172.21.15.79:6827/935518328","heartbeat_back_addr":"172.21.15.79:6831/935518328","heartbeat_front_addr":"172.21.15.79:6829/935518328","state":["exists","up"]},{"osd":11,"uuid":"c2ad9939-51b9-481e-b454-8cfc7451065b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6825","nonce":1087893461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6827","nonce":1087893461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6831","nonce":1087893461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6829","nonce":1087893461}]},"public_addr":"172.21.15.97:6825/1087893461","cluster_addr":"172.21.15.97:6827/1087893461","heartbeat_back_addr":"172.21.15.97:6831/1087893461","heartbeat_front_addr":"172.21.15.97:6829/1087893461","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:16.085186+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:37.523233+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:58.760655+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:45:20.737876+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:45:41.148757+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:02.139676+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:23.295222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:43.460985+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:04.429151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:25.761472+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:46.048313+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:48:07.381968+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3795991287":"2023-07-21T17:42:52.746152+0000","172.21.15.49:0/3777764053":"2023-07-21T17:42:52.746152+0000","172.21.15.49:6801/752420140":"2023-07-21T17:42:52.746152+0000","172.21.15.49:0/2521746017":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/1021575989":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/3362614128":"2023-07-21T17:42:52.746152+0000","172.21.15.49:6801/3017374204":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/1674833015":"2023-07-21T17:42:26.351231+0000","172.21.15.49:6800/3017374204":"2023-07-21T17:42:26.351231+0000","172.21.15.49:6800/752420140":"2023-07-21T17:42:52.746152+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T17:48:47.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 -- 172.21.15.49:0/3552445621 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa0b80a11d0 msgr2=0x7fa0b80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa0b80a11d0 0x7fa0b80a3690 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00feb80 tx=0x7fa0c400b410 comp rx=0 tx=0).stop 2023-07-20T17:48:47.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 -- 172.21.15.49:0/3552445621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d00fee60 msgr2=0x7fa0d0195440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d00fee60 0x7fa0d0195440 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00095c0 tx=0x7fa0c001a840 comp rx=0 tx=0).stop 2023-07-20T17:48:47.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 -- 172.21.15.49:0/3552445621 shutdown_connections 2023-07-20T17:48:47.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fa0b80a11d0 0x7fa0b80a3690 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0d0100a00 0x7fa0d019c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d00fee60 0x7fa0d0195440 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 --2- 172.21.15.49:0/3552445621 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa0d0100040 0x7fa0d0195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 -- 172.21.15.49:0/3552445621 >> 172.21.15.49:0/3552445621 conn(0x7fa0d00fa3e0 msgr2=0x7fa0d00fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:47.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 -- 172.21.15.49:0/3552445621 shutdown_connections 2023-07-20T17:48:47.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.211+0000 7fa0d588f700 1 -- 172.21.15.49:0/3552445621 wait complete. 2023-07-20T17:48:47.228 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T17:45:01.297180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T17:48:47.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T17:48:47.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.362+0000 7f6c129f0700 1 -- 172.21.15.49:0/2129625224 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c0c100850 msgr2=0x7f6c0c100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.362+0000 7f6c129f0700 1 --2- 172.21.15.49:0/2129625224 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c0c100850 0x7f6c0c100c70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0099c0 tx=0x7f6bfc009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.362+0000 7f6c129f0700 1 -- 172.21.15.49:0/2129625224 shutdown_connections 2023-07-20T17:48:47.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.362+0000 7f6c129f0700 1 --2- 172.21.15.49:0/2129625224 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c0c1023e0 0x7f6c0c108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.362+0000 7f6c129f0700 1 --2- 172.21.15.49:0/2129625224 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c0c100850 0x7f6c0c100c70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.362+0000 7f6c129f0700 1 --2- 172.21.15.49:0/2129625224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c0c101a40 0x7f6c0c101ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.362+0000 7f6c129f0700 1 -- 172.21.15.49:0/2129625224 >> 172.21.15.49:0/2129625224 conn(0x7f6c0c0fbde0 msgr2=0x7f6c0c0fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:47.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.362+0000 7f6c129f0700 1 -- 172.21.15.49:0/2129625224 shutdown_connections 2023-07-20T17:48:47.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.362+0000 7f6c129f0700 1 -- 172.21.15.49:0/2129625224 wait complete. 2023-07-20T17:48:47.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.362+0000 7f6c129f0700 1 Processor -- start 2023-07-20T17:48:47.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.362+0000 7f6c129f0700 1 -- start start 2023-07-20T17:48:47.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c129f0700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c0c100850 0x7f6c0c1965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:47.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c129f0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c0c101a40 0x7f6c0c196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:47.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c129f0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c0c1023e0 0x7f6c0c19db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:47.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c129f0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c19e130 con 0x7f6c0c1023e0 2023-07-20T17:48:47.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c129f0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c19e270 con 0x7f6c0c101a40 2023-07-20T17:48:47.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c129f0700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c19e400 con 0x7f6c0c100850 2023-07-20T17:48:47.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c10f8d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c0c1023e0 0x7f6c0c19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:47.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c10f8d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c0c1023e0 0x7f6c0c19db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36398/0 (socket says 172.21.15.49:36398) 2023-07-20T17:48:47.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c0bfff700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c0c100850 0x7f6c0c1965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:47.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c10f8d700 1 -- 172.21.15.49:0/3345516 learned_addr learned my addr 172.21.15.49:0/3345516 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:47.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c0b7fe700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c0c101a40 0x7f6c0c196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:47.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c10f8d700 1 -- 172.21.15.49:0/3345516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c0c100850 msgr2=0x7f6c0c1965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c10f8d700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c0c100850 0x7f6c0c1965b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c10f8d700 1 -- 172.21.15.49:0/3345516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c0c101a40 msgr2=0x7f6c0c196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c10f8d700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c0c101a40 0x7f6c0c196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c10f8d700 1 -- 172.21.15.49:0/3345516 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bfc0095f0 con 0x7f6c0c1023e0 2023-07-20T17:48:47.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c0b7fe700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c0c101a40 0x7f6c0c196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T17:48:47.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.363+0000 7f6c10f8d700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c0c1023e0 0x7f6c0c19db60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6c04007ee0 tx=0x7f6c0400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:47.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.364+0000 7f6c097fa700 1 -- 172.21.15.49:0/3345516 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0400dd20 con 0x7f6c0c1023e0 2023-07-20T17:48:47.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.364+0000 7f6c097fa700 1 -- 172.21.15.49:0/3345516 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c0400f040 con 0x7f6c0c1023e0 2023-07-20T17:48:47.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.364+0000 7f6c097fa700 1 -- 172.21.15.49:0/3345516 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c040048a0 con 0x7f6c0c1023e0 2023-07-20T17:48:47.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.364+0000 7f6c129f0700 1 -- 172.21.15.49:0/3345516 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0c19e6e0 con 0x7f6c0c1023e0 2023-07-20T17:48:47.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.364+0000 7f6c129f0700 1 -- 172.21.15.49:0/3345516 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c19ebf0 con 0x7f6c0c1023e0 2023-07-20T17:48:47.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.365+0000 7f6c129f0700 1 -- 172.21.15.49:0/3345516 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0c107e90 con 0x7f6c0c1023e0 2023-07-20T17:48:47.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.366+0000 7f6c097fa700 1 -- 172.21.15.49:0/3345516 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6c04003a00 con 0x7f6c0c1023e0 2023-07-20T17:48:47.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.366+0000 7f6c097fa700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6bf40a11d0 0x7f6bf40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:47.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.367+0000 7f6c097fa700 1 -- 172.21.15.49:0/3345516 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6c040cbb60 con 0x7f6c0c1023e0 2023-07-20T17:48:47.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.367+0000 7f6c0bfff700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6bf40a11d0 0x7f6bf40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:47.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.367+0000 7f6c0bfff700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6bf40a11d0 0x7f6bf40a3690 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc009990 tx=0x7f6bfc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:47.374 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.371+0000 7f6c097fa700 1 -- 172.21.15.49:0/3345516 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6c0404e780 con 0x7f6c0c1023e0 2023-07-20T17:48:47.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:47 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1346786036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:47.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:47 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3552445621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:47.596 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.593+0000 7f6c129f0700 1 -- 172.21.15.49:0/3345516 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c0c02c970 con 0x7f6c0c1023e0 2023-07-20T17:48:47.596 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.594+0000 7f6c097fa700 1 -- 172.21.15.49:0/3345516 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c04095750 con 0x7f6c0c1023e0 2023-07-20T17:48:47.597 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2023-07-20T17:48:47.598 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 -- 172.21.15.49:0/3345516 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6bf40a11d0 msgr2=0x7f6bf40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6bf40a11d0 0x7f6bf40a3690 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc009990 tx=0x7f6bfc005fb0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 -- 172.21.15.49:0/3345516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c0c1023e0 msgr2=0x7f6c0c19db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c0c1023e0 0x7f6c0c19db60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6c04007ee0 tx=0x7f6c0400d3b0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 -- 172.21.15.49:0/3345516 shutdown_connections 2023-07-20T17:48:47.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f6bf40a11d0 0x7f6bf40a3690 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c0c100850 0x7f6c0c1965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.600 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c0c1023e0 0x7f6c0c19db60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.600 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 --2- 172.21.15.49:0/3345516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c0c101a40 0x7f6c0c196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.600 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 -- 172.21.15.49:0/3345516 >> 172.21.15.49:0/3345516 conn(0x7f6c0c0fbde0 msgr2=0x7f6c0c104fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:47.600 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 -- 172.21.15.49:0/3345516 shutdown_connections 2023-07-20T17:48:47.600 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.596+0000 7f6c129f0700 1 -- 172.21.15.49:0/3345516 wait complete. 2023-07-20T17:48:47.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-20T17:48:47.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.738+0000 7f7e851d6700 1 -- 172.21.15.49:0/2967197894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e800ff760 msgr2=0x7f7e800ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.738+0000 7f7e851d6700 1 --2- 172.21.15.49:0/2967197894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e800ff760 0x7f7e800ffb80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7e700099c0 tx=0x7f7e70009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.738+0000 7f7e851d6700 1 -- 172.21.15.49:0/2967197894 shutdown_connections 2023-07-20T17:48:47.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.738+0000 7f7e851d6700 1 --2- 172.21.15.49:0/2967197894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e801012f0 0x7f7e80107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.738+0000 7f7e851d6700 1 --2- 172.21.15.49:0/2967197894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e800ff760 0x7f7e800ffb80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.738+0000 7f7e851d6700 1 --2- 172.21.15.49:0/2967197894 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e80100950 0x7f7e80100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.738+0000 7f7e851d6700 1 -- 172.21.15.49:0/2967197894 >> 172.21.15.49:0/2967197894 conn(0x7f7e800facd0 msgr2=0x7f7e800fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:47.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.738+0000 7f7e851d6700 1 -- 172.21.15.49:0/2967197894 shutdown_connections 2023-07-20T17:48:47.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.738+0000 7f7e851d6700 1 -- 172.21.15.49:0/2967197894 wait complete. 2023-07-20T17:48:47.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.739+0000 7f7e851d6700 1 Processor -- start 2023-07-20T17:48:47.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.739+0000 7f7e851d6700 1 -- start start 2023-07-20T17:48:47.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.739+0000 7f7e851d6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e800ff760 0x7f7e80069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:47.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.739+0000 7f7e851d6700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e80100950 0x7f7e8019b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:47.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e851d6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e801012f0 0x7f7e8019d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:47.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e851d6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8019e040 con 0x7f7e800ff760 2023-07-20T17:48:47.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e851d6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8019e1b0 con 0x7f7e801012f0 2023-07-20T17:48:47.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e851d6700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8019e370 con 0x7f7e80100950 2023-07-20T17:48:47.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e7ed9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e800ff760 0x7f7e80069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:47.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e7ed9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e800ff760 0x7f7e80069350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36408/0 (socket says 172.21.15.49:36408) 2023-07-20T17:48:47.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e7ed9d700 1 -- 172.21.15.49:0/335524626 learned_addr learned my addr 172.21.15.49:0/335524626 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:47.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e7f59e700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e801012f0 0x7f7e8019d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:47.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e7e59c700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e80100950 0x7f7e8019b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:47.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e7ed9d700 1 -- 172.21.15.49:0/335524626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e80100950 msgr2=0x7f7e8019b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e7ed9d700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e80100950 0x7f7e8019b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e7ed9d700 1 -- 172.21.15.49:0/335524626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e801012f0 msgr2=0x7f7e8019d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e7ed9d700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e801012f0 0x7f7e8019d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e7ed9d700 1 -- 172.21.15.49:0/335524626 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e700095f0 con 0x7f7e800ff760 2023-07-20T17:48:47.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e7ed9d700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e800ff760 0x7f7e80069350 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009990 tx=0x7f7e70005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:47.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e67fff700 1 -- 172.21.15.49:0/335524626 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7001d070 con 0x7f7e800ff760 2023-07-20T17:48:47.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e67fff700 1 -- 172.21.15.49:0/335524626 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e70004200 con 0x7f7e800ff760 2023-07-20T17:48:47.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e851d6700 1 -- 172.21.15.49:0/335524626 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8019e5f0 con 0x7f7e800ff760 2023-07-20T17:48:47.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e67fff700 1 -- 172.21.15.49:0/335524626 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e70022e30 con 0x7f7e800ff760 2023-07-20T17:48:47.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.740+0000 7f7e851d6700 1 -- 172.21.15.49:0/335524626 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8019eaa0 con 0x7f7e800ff760 2023-07-20T17:48:47.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.741+0000 7f7e851d6700 1 -- 172.21.15.49:0/335524626 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8010af70 con 0x7f7e800ff760 2023-07-20T17:48:47.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.742+0000 7f7e67fff700 1 -- 172.21.15.49:0/335524626 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7e70003d40 con 0x7f7e800ff760 2023-07-20T17:48:47.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.743+0000 7f7e67fff700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f7e680a1150 0x7f7e680a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:47.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.743+0000 7f7e67fff700 1 -- 172.21.15.49:0/335524626 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7e700cd6f0 con 0x7f7e800ff760 2023-07-20T17:48:47.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.743+0000 7f7e7e59c700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f7e680a1150 0x7f7e680a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:47.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.743+0000 7f7e7e59c700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f7e680a1150 0x7f7e680a3610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7e74005950 tx=0x7f7e740058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:47.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.747+0000 7f7e67fff700 1 -- 172.21.15.49:0/335524626 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7e70097370 con 0x7f7e800ff760 2023-07-20T17:48:47.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:47 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1346786036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:47.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:47 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3552445621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:47.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:47 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1346786036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T17:48:47.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:47 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3552445621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:47.957 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.954+0000 7f7e851d6700 1 -- 172.21.15.49:0/335524626 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7e8019f0e0 con 0x7f7e800ff760 2023-07-20T17:48:47.957 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.955+0000 7f7e67fff700 1 -- 172.21.15.49:0/335524626 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7e7009a990 con 0x7f7e800ff760 2023-07-20T17:48:47.958 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:47.958 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2023-07-20T17:48:47.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2023-07-20T17:48:47.959 INFO:teuthology.orchestra.run.smithi049.stdout: "poolnum": 1, 2023-07-20T17:48:47.959 INFO:teuthology.orchestra.run.smithi049.stdout: "poolname": ".mgr" 2023-07-20T17:48:47.959 INFO:teuthology.orchestra.run.smithi049.stdout: } 2023-07-20T17:48:47.959 INFO:teuthology.orchestra.run.smithi049.stdout:] 2023-07-20T17:48:47.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 -- 172.21.15.49:0/335524626 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f7e680a1150 msgr2=0x7f7e680a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.961 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f7e680a1150 0x7f7e680a3610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7e74005950 tx=0x7f7e740058e0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.961 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 -- 172.21.15.49:0/335524626 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e800ff760 msgr2=0x7f7e80069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:47.961 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e800ff760 0x7f7e80069350 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009990 tx=0x7f7e70005e80 comp rx=0 tx=0).stop 2023-07-20T17:48:47.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 -- 172.21.15.49:0/335524626 shutdown_connections 2023-07-20T17:48:47.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f7e680a1150 0x7f7e680a3610 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e801012f0 0x7f7e8019d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e800ff760 0x7f7e80069350 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 --2- 172.21.15.49:0/335524626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e80100950 0x7f7e8019b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:47.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 -- 172.21.15.49:0/335524626 >> 172.21.15.49:0/335524626 conn(0x7f7e800facd0 msgr2=0x7f7e800fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:47.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 -- 172.21.15.49:0/335524626 shutdown_connections 2023-07-20T17:48:47.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:47.957+0000 7f7e851d6700 1 -- 172.21.15.49:0/335524626 wait complete. 2023-07-20T17:48:47.973 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-20T17:48:47.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-20T17:48:48.105 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.102+0000 7f247f2c0700 1 -- 172.21.15.49:0/1524090733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2478100850 msgr2=0x7f2478100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:48.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.102+0000 7f247f2c0700 1 --2- 172.21.15.49:0/1524090733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2478100850 0x7f2478100c70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f247400b0d0 tx=0x7f247400b3e0 comp rx=0 tx=0).stop 2023-07-20T17:48:48.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.102+0000 7f247f2c0700 1 -- 172.21.15.49:0/1524090733 shutdown_connections 2023-07-20T17:48:48.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.102+0000 7f247f2c0700 1 --2- 172.21.15.49:0/1524090733 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f24781023e0 0x7f2478108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:48.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.102+0000 7f247f2c0700 1 --2- 172.21.15.49:0/1524090733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2478100850 0x7f2478100c70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:48.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.102+0000 7f247f2c0700 1 --2- 172.21.15.49:0/1524090733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2478101a40 0x7f2478101ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:48.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.102+0000 7f247f2c0700 1 -- 172.21.15.49:0/1524090733 >> 172.21.15.49:0/1524090733 conn(0x7f24780fbde0 msgr2=0x7f24780fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:48.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.102+0000 7f247f2c0700 1 -- 172.21.15.49:0/1524090733 shutdown_connections 2023-07-20T17:48:48.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.102+0000 7f247f2c0700 1 -- 172.21.15.49:0/1524090733 wait complete. 2023-07-20T17:48:48.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.103+0000 7f247f2c0700 1 Processor -- start 2023-07-20T17:48:48.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.103+0000 7f247f2c0700 1 -- start start 2023-07-20T17:48:48.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.103+0000 7f247f2c0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2478100850 0x7f2478196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:48.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.103+0000 7f247f2c0700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2478101a40 0x7f2478196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:48.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247f2c0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f24781023e0 0x7f247819dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:48.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247f2c0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247819e210 con 0x7f2478100850 2023-07-20T17:48:48.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247f2c0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247819e350 con 0x7f24781023e0 2023-07-20T17:48:48.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247f2c0700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247819e4e0 con 0x7f2478101a40 2023-07-20T17:48:48.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247d05c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2478100850 0x7f2478196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:48.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247c85b700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2478101a40 0x7f2478196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:48.110 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247d05c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2478100850 0x7f2478196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36422/0 (socket says 172.21.15.49:36422) 2023-07-20T17:48:48.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247d05c700 1 -- 172.21.15.49:0/110103244 learned_addr learned my addr 172.21.15.49:0/110103244 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:48.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247d85d700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f24781023e0 0x7f247819dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:48.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247d05c700 1 -- 172.21.15.49:0/110103244 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2478101a40 msgr2=0x7f2478196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:48.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247d05c700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2478101a40 0x7f2478196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:48.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247d05c700 1 -- 172.21.15.49:0/110103244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f24781023e0 msgr2=0x7f247819dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:48.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247d05c700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f24781023e0 0x7f247819dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:48.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247d05c700 1 -- 172.21.15.49:0/110103244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2470009520 con 0x7f2478100850 2023-07-20T17:48:48.111 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247c85b700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2478101a40 0x7f2478196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T17:48:48.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247d05c700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2478100850 0x7f2478196690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2474000f80 tx=0x7f2474008ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:48.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f246e7fc700 1 -- 172.21.15.49:0/110103244 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247400e040 con 0x7f2478100850 2023-07-20T17:48:48.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f246e7fc700 1 -- 172.21.15.49:0/110103244 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f247400be30 con 0x7f2478100850 2023-07-20T17:48:48.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247f2c0700 1 -- 172.21.15.49:0/110103244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2474009d00 con 0x7f2478100850 2023-07-20T17:48:48.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f246e7fc700 1 -- 172.21.15.49:0/110103244 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247401ba00 con 0x7f2478100850 2023-07-20T17:48:48.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.104+0000 7f247f2c0700 1 -- 172.21.15.49:0/110103244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247819ea50 con 0x7f2478100850 2023-07-20T17:48:48.112 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.106+0000 7f247f2c0700 1 -- 172.21.15.49:0/110103244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24781077d0 con 0x7f2478100850 2023-07-20T17:48:48.113 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.107+0000 7f246e7fc700 1 -- 172.21.15.49:0/110103244 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f247401bb60 con 0x7f2478100850 2023-07-20T17:48:48.113 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.108+0000 7f246e7fc700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f24640a1150 0x7f24640a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:48.114 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.109+0000 7f247c85b700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f24640a1150 0x7f24640a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:48.114 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.109+0000 7f246e7fc700 1 -- 172.21.15.49:0/110103244 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f24740cdc40 con 0x7f2478100850 2023-07-20T17:48:48.114 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.109+0000 7f247c85b700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f24640a1150 0x7f24640a3610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2468011490 tx=0x7f2468009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:48.114 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.112+0000 7f246e7fc700 1 -- 172.21.15.49:0/110103244 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f24740978c0 con 0x7f2478100850 2023-07-20T17:48:48.345 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:48.342+0000 7f247f2c0700 1 -- 172.21.15.49:0/110103244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f247802c970 con 0x7f2478100850 2023-07-20T17:48:49.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:48 smithi079 ceph-mon[98036]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:49.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:48 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3345516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T17:48:49.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:48 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:49.068 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:48 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/335524626' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T17:48:49.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:48 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:49.069 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:48 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:49.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:48 smithi049 ceph-mon[96530]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:49.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:48 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3345516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T17:48:49.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:48 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:49.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:48 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/335524626' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T17:48:49.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:48 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:49.241 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:48 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:49.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:48 smithi097 ceph-mon[98398]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:49.268 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:48 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3345516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T17:48:49.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:48 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:49.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:48 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/335524626' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T17:48:49.269 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:48 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:49.270 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:48 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:49.314 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.311+0000 7f246e7fc700 1 -- 172.21.15.49:0/110103244 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7f247409aee0 con 0x7f2478100850 2023-07-20T17:48:49.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.315+0000 7f247f2c0700 1 -- 172.21.15.49:0/110103244 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f24640a1150 msgr2=0x7f24640a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:49.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.315+0000 7f247f2c0700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f24640a1150 0x7f24640a3610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2468011490 tx=0x7f2468009260 comp rx=0 tx=0).stop 2023-07-20T17:48:49.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.315+0000 7f247f2c0700 1 -- 172.21.15.49:0/110103244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2478100850 msgr2=0x7f2478196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:49.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.315+0000 7f247f2c0700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2478100850 0x7f2478196690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2474000f80 tx=0x7f2474008ec0 comp rx=0 tx=0).stop 2023-07-20T17:48:49.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.315+0000 7f247f2c0700 1 -- 172.21.15.49:0/110103244 shutdown_connections 2023-07-20T17:48:49.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.315+0000 7f247f2c0700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f24640a1150 0x7f24640a3610 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:49.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.315+0000 7f247f2c0700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f24781023e0 0x7f247819dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:49.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.315+0000 7f247f2c0700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2478100850 0x7f2478196690 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:49.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.315+0000 7f247f2c0700 1 --2- 172.21.15.49:0/110103244 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2478101a40 0x7f2478196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:49.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.315+0000 7f247f2c0700 1 -- 172.21.15.49:0/110103244 >> 172.21.15.49:0/110103244 conn(0x7f24780fbde0 msgr2=0x7f24780fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:49.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.316+0000 7f247f2c0700 1 -- 172.21.15.49:0/110103244 shutdown_connections 2023-07-20T17:48:49.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.316+0000 7f247f2c0700 1 -- 172.21.15.49:0/110103244 wait complete. 2023-07-20T17:48:49.322 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'cephfs_metadata' created 2023-07-20T17:48:49.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-07-20T17:48:49.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.464+0000 7fdd57e43700 1 -- 172.21.15.49:0/3525776292 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd50100930 msgr2=0x7fdd50100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:49.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.464+0000 7fdd57e43700 1 --2- 172.21.15.49:0/3525776292 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd50100930 0x7fdd50100d90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0099c0 tx=0x7fdd4c009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:49.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.464+0000 7fdd57e43700 1 -- 172.21.15.49:0/3525776292 shutdown_connections 2023-07-20T17:48:49.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.464+0000 7fdd57e43700 1 --2- 172.21.15.49:0/3525776292 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd501012d0 0x7fdd50107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:49.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.464+0000 7fdd57e43700 1 --2- 172.21.15.49:0/3525776292 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd500ff740 0x7fdd500ffb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:49.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.464+0000 7fdd57e43700 1 --2- 172.21.15.49:0/3525776292 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd50100930 0x7fdd50100d90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:49.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.464+0000 7fdd57e43700 1 -- 172.21.15.49:0/3525776292 >> 172.21.15.49:0/3525776292 conn(0x7fdd500facf0 msgr2=0x7fdd500fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:49.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.464+0000 7fdd57e43700 1 -- 172.21.15.49:0/3525776292 shutdown_connections 2023-07-20T17:48:49.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.464+0000 7fdd57e43700 1 -- 172.21.15.49:0/3525776292 wait complete. 2023-07-20T17:48:49.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd57e43700 1 Processor -- start 2023-07-20T17:48:49.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd57e43700 1 -- start start 2023-07-20T17:48:49.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd57e43700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd500ff740 0x7fdd501965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:49.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd57e43700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd50100930 0x7fdd50196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:49.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd57e43700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd501012d0 0x7fdd5019db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:49.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd57e43700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5019e130 con 0x7fdd500ff740 2023-07-20T17:48:49.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd57e43700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5019e270 con 0x7fdd50100930 2023-07-20T17:48:49.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd57e43700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5019e400 con 0x7fdd501012d0 2023-07-20T17:48:49.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd55bdf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd500ff740 0x7fdd501965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:49.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd55bdf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd500ff740 0x7fdd501965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36436/0 (socket says 172.21.15.49:36436) 2023-07-20T17:48:49.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd55bdf700 1 -- 172.21.15.49:0/1702119184 learned_addr learned my addr 172.21.15.49:0/1702119184 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:49.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd553de700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd50100930 0x7fdd50196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:49.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd563e0700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd501012d0 0x7fdd5019db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:49.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd55bdf700 1 -- 172.21.15.49:0/1702119184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd501012d0 msgr2=0x7fdd5019db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:49.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd55bdf700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd501012d0 0x7fdd5019db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:49.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd55bdf700 1 -- 172.21.15.49:0/1702119184 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd50100930 msgr2=0x7fdd50196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:49.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd55bdf700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd50100930 0x7fdd50196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:49.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.465+0000 7fdd55bdf700 1 -- 172.21.15.49:0/1702119184 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c0095f0 con 0x7fdd500ff740 2023-07-20T17:48:49.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.466+0000 7fdd55bdf700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd500ff740 0x7fdd501965b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000d980 tx=0x7fdd4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:49.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.466+0000 7fdd46ffd700 1 -- 172.21.15.49:0/1702119184 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40004020 con 0x7fdd500ff740 2023-07-20T17:48:49.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.466+0000 7fdd46ffd700 1 -- 172.21.15.49:0/1702119184 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd4000f690 con 0x7fdd500ff740 2023-07-20T17:48:49.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.466+0000 7fdd57e43700 1 -- 172.21.15.49:0/1702119184 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5019e6e0 con 0x7fdd500ff740 2023-07-20T17:48:49.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.466+0000 7fdd46ffd700 1 -- 172.21.15.49:0/1702119184 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40003a60 con 0x7fdd500ff740 2023-07-20T17:48:49.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.466+0000 7fdd57e43700 1 -- 172.21.15.49:0/1702119184 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5019ebf0 con 0x7fdd500ff740 2023-07-20T17:48:49.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.467+0000 7fdd57e43700 1 -- 172.21.15.49:0/1702119184 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd50106d00 con 0x7fdd500ff740 2023-07-20T17:48:49.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.468+0000 7fdd46ffd700 1 -- 172.21.15.49:0/1702119184 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fdd4000fb00 con 0x7fdd500ff740 2023-07-20T17:48:49.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.468+0000 7fdd46ffd700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fdd3c0a1150 0x7fdd3c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:49.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.469+0000 7fdd46ffd700 1 -- 172.21.15.49:0/1702119184 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fdd400cc1b0 con 0x7fdd500ff740 2023-07-20T17:48:49.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.469+0000 7fdd553de700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fdd3c0a1150 0x7fdd3c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:49.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.469+0000 7fdd553de700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fdd3c0a1150 0x7fdd3c0a3610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00b580 tx=0x7fdd4c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:49.477 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.473+0000 7fdd46ffd700 1 -- 172.21.15.49:0/1702119184 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdd40095c30 con 0x7fdd500ff740 2023-07-20T17:48:49.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:49.690+0000 7fdd57e43700 1 -- 172.21.15.49:0/1702119184 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdd5002c970 con 0x7fdd500ff740 2023-07-20T17:48:50.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.314+0000 7fdd46ffd700 1 -- 172.21.15.49:0/1702119184 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) v1 ==== 170+0+0 (secure 0 0 0) 0x7fdd40099250 con 0x7fdd500ff740 2023-07-20T17:48:50.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 -- 172.21.15.49:0/1702119184 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fdd3c0a1150 msgr2=0x7fdd3c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fdd3c0a1150 0x7fdd3c0a3610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00b580 tx=0x7fdd4c005fb0 comp rx=0 tx=0).stop 2023-07-20T17:48:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 -- 172.21.15.49:0/1702119184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd500ff740 msgr2=0x7fdd501965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd500ff740 0x7fdd501965b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000d980 tx=0x7fdd4000dc90 comp rx=0 tx=0).stop 2023-07-20T17:48:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 -- 172.21.15.49:0/1702119184 shutdown_connections 2023-07-20T17:48:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fdd3c0a1150 0x7fdd3c0a3610 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdd501012d0 0x7fdd5019db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd500ff740 0x7fdd501965b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 --2- 172.21.15.49:0/1702119184 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd50100930 0x7fdd50196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 -- 172.21.15.49:0/1702119184 >> 172.21.15.49:0/1702119184 conn(0x7fdd500facf0 msgr2=0x7fdd50103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:50.327 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 -- 172.21.15.49:0/1702119184 shutdown_connections 2023-07-20T17:48:50.327 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.322+0000 7fdd57e43700 1 -- 172.21.15.49:0/1702119184 wait complete. 2023-07-20T17:48:50.327 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'cephfs_data' created 2023-07-20T17:48:50.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-20T17:48:50.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.480+0000 7f3820809700 1 -- 172.21.15.49:0/241335053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f381c106560 msgr2=0x7f381c108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:50.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.480+0000 7f3820809700 1 --2- 172.21.15.49:0/241335053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f381c106560 0x7f381c108970 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f381400b0a0 tx=0x7f381400b3b0 comp rx=0 tx=0).stop 2023-07-20T17:48:50.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.480+0000 7f3820809700 1 -- 172.21.15.49:0/241335053 shutdown_connections 2023-07-20T17:48:50.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.480+0000 7f3820809700 1 --2- 172.21.15.49:0/241335053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f381c101300 0x7f381c1036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:50.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.480+0000 7f3820809700 1 --2- 172.21.15.49:0/241335053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f381c106560 0x7f381c108970 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:50.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.480+0000 7f3820809700 1 --2- 172.21.15.49:0/241335053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f381c103c30 0x7f381c106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:50.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.480+0000 7f3820809700 1 -- 172.21.15.49:0/241335053 >> 172.21.15.49:0/241335053 conn(0x7f381c0fac70 msgr2=0x7f381c0fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:50.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.480+0000 7f3820809700 1 -- 172.21.15.49:0/241335053 shutdown_connections 2023-07-20T17:48:50.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.480+0000 7f3820809700 1 -- 172.21.15.49:0/241335053 wait complete. 2023-07-20T17:48:50.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f3820809700 1 Processor -- start 2023-07-20T17:48:50.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f3820809700 1 -- start start 2023-07-20T17:48:50.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f3820809700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f381c101300 0x7f381c100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:50.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f3820809700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f381c103c30 0x7f381c0ff2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:50.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f3820809700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f381c106560 0x7f381c0ff7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:50.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f3820809700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0fff90 con 0x7f381c106560 2023-07-20T17:48:50.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f3820809700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c100100 con 0x7f381c103c30 2023-07-20T17:48:50.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f3820809700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c1a0660 con 0x7f381c101300 2023-07-20T17:48:50.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f381ad9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f381c106560 0x7f381c0ff7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:50.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f3819d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f381c103c30 0x7f381c0ff2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:50.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f381ad9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f381c106560 0x7f381c0ff7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36458/0 (socket says 172.21.15.49:36458) 2023-07-20T17:48:50.488 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f381a59c700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f381c101300 0x7f381c100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:50.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f381ad9d700 1 -- 172.21.15.49:0/1933907616 learned_addr learned my addr 172.21.15.49:0/1933907616 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:50.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f381ad9d700 1 -- 172.21.15.49:0/1933907616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f381c101300 msgr2=0x7f381c100c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:50.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f381ad9d700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f381c101300 0x7f381c100c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:50.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f381ad9d700 1 -- 172.21.15.49:0/1933907616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f381c103c30 msgr2=0x7f381c0ff2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:50.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f381ad9d700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f381c103c30 0x7f381c0ff2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:50.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f381ad9d700 1 -- 172.21.15.49:0/1933907616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3814009d00 con 0x7f381c106560 2023-07-20T17:48:50.489 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.481+0000 7f381a59c700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f381c101300 0x7f381c100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:48:50.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.482+0000 7f381ad9d700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f381c106560 0x7f381c0ff7f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f381400b780 tx=0x7f381400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:50.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.482+0000 7f380b7fe700 1 -- 172.21.15.49:0/1933907616 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381400e040 con 0x7f381c106560 2023-07-20T17:48:50.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.482+0000 7f3820809700 1 -- 172.21.15.49:0/1933907616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381c1a08e0 con 0x7f381c106560 2023-07-20T17:48:50.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.482+0000 7f380b7fe700 1 -- 172.21.15.49:0/1933907616 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38140094a0 con 0x7f381c106560 2023-07-20T17:48:50.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.482+0000 7f380b7fe700 1 -- 172.21.15.49:0/1933907616 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3814005330 con 0x7f381c106560 2023-07-20T17:48:50.490 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.482+0000 7f3820809700 1 -- 172.21.15.49:0/1933907616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c1a0db0 con 0x7f381c106560 2023-07-20T17:48:50.491 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.484+0000 7f3820809700 1 -- 172.21.15.49:0/1933907616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381c0fc7f0 con 0x7f381c106560 2023-07-20T17:48:50.491 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.484+0000 7f380b7fe700 1 -- 172.21.15.49:0/1933907616 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f381401da30 con 0x7f381c106560 2023-07-20T17:48:50.491 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.485+0000 7f380b7fe700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f38040a1150 0x7f38040a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:50.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.485+0000 7f380b7fe700 1 -- 172.21.15.49:0/1933907616 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f38140cddf0 con 0x7f381c106560 2023-07-20T17:48:50.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.485+0000 7f381a59c700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f38040a1150 0x7f38040a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:50.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.485+0000 7f381a59c700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f38040a1150 0x7f38040a3610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f380c009cb0 tx=0x7f380c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:50.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:50 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/110103244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T17:48:50.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:50 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/110103244' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T17:48:50.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:50 smithi049 ceph-mon[96530]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T17:48:50.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:50.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:50.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:50 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:50.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.490+0000 7f380b7fe700 1 -- 172.21.15.49:0/1933907616 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3814097780 con 0x7f381c106560 2023-07-20T17:48:50.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:50 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/110103244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T17:48:50.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:50 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/110103244' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T17:48:50.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:50 smithi097 ceph-mon[98398]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T17:48:50.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:50.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:50.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:50 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:50.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:50 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/110103244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T17:48:50.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:50 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/110103244' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T17:48:50.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:50 smithi079 ceph-mon[98036]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T17:48:50.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:50.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:50.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:50.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:50 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T17:48:50.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:50.747+0000 7f3820809700 1 -- 172.21.15.49:0/1933907616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f381c02c970 con 0x7f381c106560 2023-07-20T17:48:51.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.317+0000 7f380b7fe700 1 -- 172.21.15.49:0/1933907616 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f381409ada0 con 0x7f381c106560 2023-07-20T17:48:51.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 -- 172.21.15.49:0/1933907616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f38040a1150 msgr2=0x7f38040a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:51.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f38040a1150 0x7f38040a3610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f380c009cb0 tx=0x7f380c009260 comp rx=0 tx=0).stop 2023-07-20T17:48:51.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 -- 172.21.15.49:0/1933907616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f381c106560 msgr2=0x7f381c0ff7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:51.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f381c106560 0x7f381c0ff7f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f381400b780 tx=0x7f381400bd60 comp rx=0 tx=0).stop 2023-07-20T17:48:51.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 -- 172.21.15.49:0/1933907616 shutdown_connections 2023-07-20T17:48:51.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f38040a1150 0x7f38040a3610 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f381c101300 0x7f381c100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f381c106560 0x7f381c0ff7f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 --2- 172.21.15.49:0/1933907616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f381c103c30 0x7f381c0ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 -- 172.21.15.49:0/1933907616 >> 172.21.15.49:0/1933907616 conn(0x7f381c0fac70 msgr2=0x7f381c101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:51.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 -- 172.21.15.49:0/1933907616 shutdown_connections 2023-07-20T17:48:51.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.320+0000 7f3820809700 1 -- 172.21.15.49:0/1933907616 wait complete. 2023-07-20T17:48:51.328 INFO:teuthology.orchestra.run.smithi049.stderr:new fs with metadata pool 2 and data pool 3 2023-07-20T17:48:51.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T17:48:51.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.466+0000 7f49900a6700 1 -- 172.21.15.49:0/3492415084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49880ff720 msgr2=0x7f49880ffb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:51.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.466+0000 7f49900a6700 1 --2- 172.21.15.49:0/3492415084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49880ff720 0x7f49880ffb40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f49840099c0 tx=0x7f4984009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.466+0000 7f49900a6700 1 -- 172.21.15.49:0/3492415084 shutdown_connections 2023-07-20T17:48:51.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.466+0000 7f49900a6700 1 --2- 172.21.15.49:0/3492415084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f49881012b0 0x7f4988107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.466+0000 7f49900a6700 1 --2- 172.21.15.49:0/3492415084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49880ff720 0x7f49880ffb40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.466+0000 7f49900a6700 1 --2- 172.21.15.49:0/3492415084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4988100910 0x7f4988100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.466+0000 7f49900a6700 1 -- 172.21.15.49:0/3492415084 >> 172.21.15.49:0/3492415084 conn(0x7f49880facb0 msgr2=0x7f49880fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:51.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.466+0000 7f49900a6700 1 -- 172.21.15.49:0/3492415084 shutdown_connections 2023-07-20T17:48:51.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.466+0000 7f49900a6700 1 -- 172.21.15.49:0/3492415084 wait complete. 2023-07-20T17:48:51.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f49900a6700 1 Processor -- start 2023-07-20T17:48:51.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f49900a6700 1 -- start start 2023-07-20T17:48:51.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f49900a6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49880ff720 0x7f49880685b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:51.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f49900a6700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4988100910 0x7f498819b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:51.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f49900a6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49881012b0 0x7f498819d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:51.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f49900a6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498819e020 con 0x7f49880ff720 2023-07-20T17:48:51.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f49900a6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498819e190 con 0x7f49881012b0 2023-07-20T17:48:51.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f49900a6700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498819e350 con 0x7f4988100910 2023-07-20T17:48:51.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f498de42700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49880ff720 0x7f49880685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:51.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f498de42700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49880ff720 0x7f49880685b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36472/0 (socket says 172.21.15.49:36472) 2023-07-20T17:48:51.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f498de42700 1 -- 172.21.15.49:0/1111962888 learned_addr learned my addr 172.21.15.49:0/1111962888 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:51.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f498d641700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4988100910 0x7f498819b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:51.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.467+0000 7f498e643700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49881012b0 0x7f498819d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:51.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.468+0000 7f498de42700 1 -- 172.21.15.49:0/1111962888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4988100910 msgr2=0x7f498819b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:51.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.468+0000 7f498de42700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4988100910 0x7f498819b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.468+0000 7f498de42700 1 -- 172.21.15.49:0/1111962888 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49881012b0 msgr2=0x7f498819d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:51.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.468+0000 7f498de42700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49881012b0 0x7f498819d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.468+0000 7f498de42700 1 -- 172.21.15.49:0/1111962888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49840095f0 con 0x7f49880ff720 2023-07-20T17:48:51.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.468+0000 7f498de42700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49880ff720 0x7f49880685b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4984009fd0 tx=0x7f498400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:51.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.468+0000 7f497affd700 1 -- 172.21.15.49:0/1111962888 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498401d070 con 0x7f49880ff720 2023-07-20T17:48:51.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.468+0000 7f497affd700 1 -- 172.21.15.49:0/1111962888 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4984005030 con 0x7f49880ff720 2023-07-20T17:48:51.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.468+0000 7f497affd700 1 -- 172.21.15.49:0/1111962888 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4984004aa0 con 0x7f49880ff720 2023-07-20T17:48:51.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.468+0000 7f49900a6700 1 -- 172.21.15.49:0/1111962888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498819e5d0 con 0x7f49880ff720 2023-07-20T17:48:51.477 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.468+0000 7f49900a6700 1 -- 172.21.15.49:0/1111962888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498819e9f0 con 0x7f49880ff720 2023-07-20T17:48:51.477 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.469+0000 7f49900a6700 1 -- 172.21.15.49:0/1111962888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49881066a0 con 0x7f49880ff720 2023-07-20T17:48:51.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.470+0000 7f497affd700 1 -- 172.21.15.49:0/1111962888 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4984005410 con 0x7f49880ff720 2023-07-20T17:48:51.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.471+0000 7f497affd700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f49740a11d0 0x7f49740a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:51.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.471+0000 7f498d641700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f49740a11d0 0x7f49740a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:51.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.471+0000 7f497affd700 1 -- 172.21.15.49:0/1111962888 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f49840cd9d0 con 0x7f49880ff720 2023-07-20T17:48:51.478 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.471+0000 7f498d641700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f49740a11d0 0x7f49740a3690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f497c005950 tx=0x7f497c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:51.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.474+0000 7f497affd700 1 -- 172.21.15.49:0/1111962888 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4984097260 con 0x7f49880ff720 2023-07-20T17:48:51.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:51 smithi049 ceph-mon[96530]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:51.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:51 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1702119184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T17:48:51.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:51 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1702119184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T17:48:51.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:51 smithi049 ceph-mon[96530]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T17:48:51.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:51 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1933907616' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T17:48:51.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:51 smithi049 ceph-mon[96530]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T17:48:51.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:51 smithi097 ceph-mon[98398]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:51.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:51 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1702119184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T17:48:51.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:51 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1702119184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T17:48:51.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:51 smithi097 ceph-mon[98398]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T17:48:51.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:51 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1933907616' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T17:48:51.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:51 smithi097 ceph-mon[98398]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T17:48:51.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:51 smithi079 ceph-mon[98036]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:51.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:51 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1702119184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T17:48:51.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:51 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1702119184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T17:48:51.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:51 smithi079 ceph-mon[98036]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T17:48:51.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:51 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1933907616' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T17:48:51.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:51 smithi079 ceph-mon[98036]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T17:48:51.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.689+0000 7f49900a6700 1 -- 172.21.15.49:0/1111962888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f498802c970 con 0x7f49880ff720 2023-07-20T17:48:51.694 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.691+0000 7f497affd700 1 -- 172.21.15.49:0/1111962888 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22024 (secure 0 0 0) 0x7f498409a880 con 0x7f49880ff720 2023-07-20T17:48:51.694 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:51.695 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":81,"fsid":"8c050cea-2724-11ee-9b34-001a4aab830c","created":"2023-07-20T17:41:57.467323+0000","modified":"2023-07-20T17:48:50.749432+0000","last_up_change":"2023-07-20T17:48:09.220884+0000","last_in_change":"2023-07-20T17:47:51.156799+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T17:45:01.297180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T17:48:48.782902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T17:48:50.129505+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63bfce41-a0a8-49b6-aa1e-ed7809c0627e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6803","nonce":2306317776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6805","nonce":2306317776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6809","nonce":2306317776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6807","nonce":2306317776}]},"public_addr":"172.21.15.49:6803/2306317776","cluster_addr":"172.21.15.49:6805/2306317776","heartbeat_back_addr":"172.21.15.49:6809/2306317776","heartbeat_front_addr":"172.21.15.49:6807/2306317776","state":["exists","up"]},{"osd":1,"uuid":"48cd9d65-b37d-410a-8007-f53fbd24dd73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6801","nonce":2048064928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6803","nonce":2048064928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6807","nonce":2048064928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6805","nonce":2048064928}]},"public_addr":"172.21.15.79:6801/2048064928","cluster_addr":"172.21.15.79:6803/2048064928","heartbeat_back_addr":"172.21.15.79:6807/2048064928","heartbeat_front_addr":"172.21.15.79:6805/2048064928","state":["exists","up"]},{"osd":2,"uuid":"272a17dd-e825-4990-a178-c2eb6985f37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6801","nonce":2945387274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6803","nonce":2945387274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6807","nonce":2945387274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6805","nonce":2945387274}]},"public_addr":"172.21.15.97:6801/2945387274","cluster_addr":"172.21.15.97:6803/2945387274","heartbeat_back_addr":"172.21.15.97:6807/2945387274","heartbeat_front_addr":"172.21.15.97:6805/2945387274","state":["exists","up"]},{"osd":3,"uuid":"6ba1972a-2bf4-4602-900c-11b15dcf6027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6811","nonce":178015359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6813","nonce":178015359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6817","nonce":178015359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6815","nonce":178015359}]},"public_addr":"172.21.15.49:6811/178015359","cluster_addr":"172.21.15.49:6813/178015359","heartbeat_back_addr":"172.21.15.49:6817/178015359","heartbeat_front_addr":"172.21.15.49:6815/178015359","state":["exists","up"]},{"osd":4,"uuid":"a167452b-f627-4aae-aecd-e513a5649235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6809","nonce":4037011752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6811","nonce":4037011752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6815","nonce":4037011752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6813","nonce":4037011752}]},"public_addr":"172.21.15.79:6809/4037011752","cluster_addr":"172.21.15.79:6811/4037011752","heartbeat_back_addr":"172.21.15.79:6815/4037011752","heartbeat_front_addr":"172.21.15.79:6813/4037011752","state":["exists","up"]},{"osd":5,"uuid":"def1f97c-84c4-4c87-9363-62f72142e0a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6809","nonce":2172761707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6811","nonce":2172761707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6815","nonce":2172761707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6813","nonce":2172761707}]},"public_addr":"172.21.15.97:6809/2172761707","cluster_addr":"172.21.15.97:6811/2172761707","heartbeat_back_addr":"172.21.15.97:6815/2172761707","heartbeat_front_addr":"172.21.15.97:6813/2172761707","state":["exists","up"]},{"osd":6,"uuid":"e8c200c2-0692-4d5e-a281-77c9ad6670cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6819","nonce":797242207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6821","nonce":797242207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6825","nonce":797242207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6823","nonce":797242207}]},"public_addr":"172.21.15.49:6819/797242207","cluster_addr":"172.21.15.49:6821/797242207","heartbeat_back_addr":"172.21.15.49:6825/797242207","heartbeat_front_addr":"172.21.15.49:6823/797242207","state":["exists","up"]},{"osd":7,"uuid":"847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6817","nonce":1417643292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6819","nonce":1417643292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6823","nonce":1417643292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6821","nonce":1417643292}]},"public_addr":"172.21.15.79:6817/1417643292","cluster_addr":"172.21.15.79:6819/1417643292","heartbeat_back_addr":"172.21.15.79:6823/1417643292","heartbeat_front_addr":"172.21.15.79:6821/1417643292","state":["exists","up"]},{"osd":8,"uuid":"14c55ac6-0e96-42a4-b974-178c87c2185c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6817","nonce":1465924719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6819","nonce":1465924719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6823","nonce":1465924719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6821","nonce":1465924719}]},"public_addr":"172.21.15.97:6817/1465924719","cluster_addr":"172.21.15.97:6819/1465924719","heartbeat_back_addr":"172.21.15.97:6823/1465924719","heartbeat_front_addr":"172.21.15.97:6821/1465924719","state":["exists","up"]},{"osd":9,"uuid":"b0c07e94-8039-4e41-85b7-23344124687b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6827","nonce":845176131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6829","nonce":845176131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6833","nonce":845176131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6831","nonce":845176131}]},"public_addr":"172.21.15.49:6827/845176131","cluster_addr":"172.21.15.49:6829/845176131","heartbeat_back_addr":"172.21.15.49:6833/845176131","heartbeat_front_addr":"172.21.15.49:6831/845176131","state":["exists","up"]},{"osd":10,"uuid":"67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6825","nonce":935518328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6827","nonce":935518328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6831","nonce":935518328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6829","nonce":935518328}]},"public_addr":"172.21.15.79:6825/935518328","cluster_addr":"172.21.15.79:6827/935518328","heartbeat_back_addr":"172.21.15.79:6831/935518328","heartbeat_front_addr":"172.21.15.79:6829/935518328","state":["exists","up"]},{"osd":11,"uuid":"c2ad9939-51b9-481e-b454-8cfc7451065b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6825","nonce":1087893461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6827","nonce":1087893461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6831","nonce":1087893461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6829","nonce":1087893461}]},"public_addr":"172.21.15.97:6825/1087893461","cluster_addr":"172.21.15.97:6827/1087893461","heartbeat_back_addr":"172.21.15.97:6831/1087893461","heartbeat_front_addr":"172.21.15.97:6829/1087893461","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:16.085186+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:37.523233+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:58.760655+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:45:20.737876+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:45:41.148757+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:02.139676+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:23.295222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:43.460985+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:04.429151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:25.761472+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:46.048313+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:48:07.381968+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3795991287":"2023-07-21T17:42:52.746152+0000","172.21.15.49:0/3777764053":"2023-07-21T17:42:52.746152+0000","172.21.15.49:6801/752420140":"2023-07-21T17:42:52.746152+0000","172.21.15.49:0/2521746017":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/1021575989":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/3362614128":"2023-07-21T17:42:52.746152+0000","172.21.15.49:6801/3017374204":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/1674833015":"2023-07-21T17:42:26.351231+0000","172.21.15.49:6800/3017374204":"2023-07-21T17:42:26.351231+0000","172.21.15.49:6800/752420140":"2023-07-21T17:42:52.746152+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T17:48:51.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.693+0000 7f49900a6700 1 -- 172.21.15.49:0/1111962888 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f49740a11d0 msgr2=0x7f49740a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:51.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.693+0000 7f49900a6700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f49740a11d0 0x7f49740a3690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f497c005950 tx=0x7f497c00b410 comp rx=0 tx=0).stop 2023-07-20T17:48:51.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.693+0000 7f49900a6700 1 -- 172.21.15.49:0/1111962888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49880ff720 msgr2=0x7f49880685b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:51.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.693+0000 7f49900a6700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49880ff720 0x7f49880685b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4984009fd0 tx=0x7f498400f690 comp rx=0 tx=0).stop 2023-07-20T17:48:51.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.693+0000 7f49900a6700 1 -- 172.21.15.49:0/1111962888 shutdown_connections 2023-07-20T17:48:51.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.693+0000 7f49900a6700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f49740a11d0 0x7f49740a3690 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.693+0000 7f49900a6700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49881012b0 0x7f498819d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.693+0000 7f49900a6700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49880ff720 0x7f49880685b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.693+0000 7f49900a6700 1 --2- 172.21.15.49:0/1111962888 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4988100910 0x7f498819b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.693+0000 7f49900a6700 1 -- 172.21.15.49:0/1111962888 >> 172.21.15.49:0/1111962888 conn(0x7f49880facb0 msgr2=0x7f49880fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:51.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.694+0000 7f49900a6700 1 -- 172.21.15.49:0/1111962888 shutdown_connections 2023-07-20T17:48:51.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.694+0000 7f49900a6700 1 -- 172.21.15.49:0/1111962888 wait complete. 2023-07-20T17:48:51.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T17:48:51.846 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.841+0000 7f19ae15d700 1 -- 172.21.15.49:0/4048978894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19a8103c40 msgr2=0x7f19a8106030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:51.847 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.841+0000 7f19ae15d700 1 --2- 172.21.15.49:0/4048978894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19a8103c40 0x7f19a8106030 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1990009990 tx=0x7f1990009ca0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.847 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.841+0000 7f19ae15d700 1 -- 172.21.15.49:0/4048978894 shutdown_connections 2023-07-20T17:48:51.847 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.841+0000 7f19ae15d700 1 --2- 172.21.15.49:0/4048978894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19a8101310 0x7f19a8103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.841+0000 7f19ae15d700 1 --2- 172.21.15.49:0/4048978894 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19a8106570 0x7f19a81089a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.841+0000 7f19ae15d700 1 --2- 172.21.15.49:0/4048978894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19a8103c40 0x7f19a8106030 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.841+0000 7f19ae15d700 1 -- 172.21.15.49:0/4048978894 >> 172.21.15.49:0/4048978894 conn(0x7f19a80faca0 msgr2=0x7f19a80fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:51.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.841+0000 7f19ae15d700 1 -- 172.21.15.49:0/4048978894 shutdown_connections 2023-07-20T17:48:51.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.841+0000 7f19ae15d700 1 -- 172.21.15.49:0/4048978894 wait complete. 2023-07-20T17:48:51.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.841+0000 7f19ae15d700 1 Processor -- start 2023-07-20T17:48:51.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f19ae15d700 1 -- start start 2023-07-20T17:48:51.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f19ae15d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19a8101310 0x7f19a8196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:51.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f19ae15d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19a8103c40 0x7f19a81969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:51.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f19ae15d700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19a8106570 0x7f19a819da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:51.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f19ae15d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a819dff0 con 0x7f19a8103c40 2023-07-20T17:48:51.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f19ae15d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a819e130 con 0x7f19a8101310 2023-07-20T17:48:51.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f19ae15d700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a819e2c0 con 0x7f19a8106570 2023-07-20T17:48:51.852 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f199ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19a8103c40 0x7f19a81969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:51.852 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f199ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19a8103c40 0x7f19a81969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36488/0 (socket says 172.21.15.49:36488) 2023-07-20T17:48:51.852 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f199ffff700 1 -- 172.21.15.49:0/1514087207 learned_addr learned my addr 172.21.15.49:0/1514087207 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:51.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f19a77fe700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19a8101310 0x7f19a8196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:51.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f19a7fff700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19a8106570 0x7f19a819da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:51.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f199ffff700 1 -- 172.21.15.49:0/1514087207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19a8106570 msgr2=0x7f19a819da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:51.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f199ffff700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19a8106570 0x7f19a819da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f199ffff700 1 -- 172.21.15.49:0/1514087207 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19a8101310 msgr2=0x7f19a8196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:51.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f199ffff700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19a8101310 0x7f19a8196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:51.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f199ffff700 1 -- 172.21.15.49:0/1514087207 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19900095f0 con 0x7f19a8103c40 2023-07-20T17:48:51.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f19a77fe700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19a8101310 0x7f19a8196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T17:48:51.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f199ffff700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19a8103c40 0x7f19a81969b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1990004730 tx=0x7f1990004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:51.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.842+0000 7f19a57fa700 1 -- 172.21.15.49:0/1514087207 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199001d070 con 0x7f19a8103c40 2023-07-20T17:48:51.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.843+0000 7f19a57fa700 1 -- 172.21.15.49:0/1514087207 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1990004c90 con 0x7f19a8103c40 2023-07-20T17:48:51.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.843+0000 7f19ae15d700 1 -- 172.21.15.49:0/1514087207 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a819e540 con 0x7f19a8103c40 2023-07-20T17:48:51.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.843+0000 7f19a57fa700 1 -- 172.21.15.49:0/1514087207 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19900054b0 con 0x7f19a8103c40 2023-07-20T17:48:51.857 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.843+0000 7f19ae15d700 1 -- 172.21.15.49:0/1514087207 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a819e960 con 0x7f19a8103c40 2023-07-20T17:48:51.858 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.844+0000 7f19ae15d700 1 -- 172.21.15.49:0/1514087207 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a80fc820 con 0x7f19a8103c40 2023-07-20T17:48:51.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.845+0000 7f19a57fa700 1 -- 172.21.15.49:0/1514087207 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f199000fbc0 con 0x7f19a8103c40 2023-07-20T17:48:51.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.845+0000 7f19a57fa700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f19940a11d0 0x7f19940a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:51.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.846+0000 7f19a77fe700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f19940a11d0 0x7f19940a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:51.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.846+0000 7f19a57fa700 1 -- 172.21.15.49:0/1514087207 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f19900cd990 con 0x7f19a8103c40 2023-07-20T17:48:51.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.846+0000 7f19a77fe700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f19940a11d0 0x7f19940a3690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1998005950 tx=0x7f199800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:51.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:51.850+0000 7f19a57fa700 1 -- 172.21.15.49:0/1514087207 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f199005a780 con 0x7f19a8103c40 2023-07-20T17:48:52.060 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.057+0000 7f19ae15d700 1 -- 172.21.15.49:0/1514087207 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19a802cb20 con 0x7f19a8103c40 2023-07-20T17:48:52.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.059+0000 7f19a57fa700 1 -- 172.21.15.49:0/1514087207 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22024 (secure 0 0 0) 0x7f19900972c0 con 0x7f19a8103c40 2023-07-20T17:48:52.062 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-07-20T17:48:52.063 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":81,"fsid":"8c050cea-2724-11ee-9b34-001a4aab830c","created":"2023-07-20T17:41:57.467323+0000","modified":"2023-07-20T17:48:50.749432+0000","last_up_change":"2023-07-20T17:48:09.220884+0000","last_in_change":"2023-07-20T17:47:51.156799+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T17:45:01.297180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T17:48:48.782902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T17:48:50.129505+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63bfce41-a0a8-49b6-aa1e-ed7809c0627e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6803","nonce":2306317776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6805","nonce":2306317776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6809","nonce":2306317776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2306317776},{"type":"v1","addr":"172.21.15.49:6807","nonce":2306317776}]},"public_addr":"172.21.15.49:6803/2306317776","cluster_addr":"172.21.15.49:6805/2306317776","heartbeat_back_addr":"172.21.15.49:6809/2306317776","heartbeat_front_addr":"172.21.15.49:6807/2306317776","state":["exists","up"]},{"osd":1,"uuid":"48cd9d65-b37d-410a-8007-f53fbd24dd73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6801","nonce":2048064928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6803","nonce":2048064928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6807","nonce":2048064928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2048064928},{"type":"v1","addr":"172.21.15.79:6805","nonce":2048064928}]},"public_addr":"172.21.15.79:6801/2048064928","cluster_addr":"172.21.15.79:6803/2048064928","heartbeat_back_addr":"172.21.15.79:6807/2048064928","heartbeat_front_addr":"172.21.15.79:6805/2048064928","state":["exists","up"]},{"osd":2,"uuid":"272a17dd-e825-4990-a178-c2eb6985f37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6801","nonce":2945387274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6803","nonce":2945387274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6807","nonce":2945387274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2945387274},{"type":"v1","addr":"172.21.15.97:6805","nonce":2945387274}]},"public_addr":"172.21.15.97:6801/2945387274","cluster_addr":"172.21.15.97:6803/2945387274","heartbeat_back_addr":"172.21.15.97:6807/2945387274","heartbeat_front_addr":"172.21.15.97:6805/2945387274","state":["exists","up"]},{"osd":3,"uuid":"6ba1972a-2bf4-4602-900c-11b15dcf6027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6811","nonce":178015359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6813","nonce":178015359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6817","nonce":178015359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":178015359},{"type":"v1","addr":"172.21.15.49:6815","nonce":178015359}]},"public_addr":"172.21.15.49:6811/178015359","cluster_addr":"172.21.15.49:6813/178015359","heartbeat_back_addr":"172.21.15.49:6817/178015359","heartbeat_front_addr":"172.21.15.49:6815/178015359","state":["exists","up"]},{"osd":4,"uuid":"a167452b-f627-4aae-aecd-e513a5649235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6809","nonce":4037011752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6811","nonce":4037011752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6815","nonce":4037011752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":4037011752},{"type":"v1","addr":"172.21.15.79:6813","nonce":4037011752}]},"public_addr":"172.21.15.79:6809/4037011752","cluster_addr":"172.21.15.79:6811/4037011752","heartbeat_back_addr":"172.21.15.79:6815/4037011752","heartbeat_front_addr":"172.21.15.79:6813/4037011752","state":["exists","up"]},{"osd":5,"uuid":"def1f97c-84c4-4c87-9363-62f72142e0a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6809","nonce":2172761707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6811","nonce":2172761707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6815","nonce":2172761707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2172761707},{"type":"v1","addr":"172.21.15.97:6813","nonce":2172761707}]},"public_addr":"172.21.15.97:6809/2172761707","cluster_addr":"172.21.15.97:6811/2172761707","heartbeat_back_addr":"172.21.15.97:6815/2172761707","heartbeat_front_addr":"172.21.15.97:6813/2172761707","state":["exists","up"]},{"osd":6,"uuid":"e8c200c2-0692-4d5e-a281-77c9ad6670cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6819","nonce":797242207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6821","nonce":797242207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6825","nonce":797242207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":797242207},{"type":"v1","addr":"172.21.15.49:6823","nonce":797242207}]},"public_addr":"172.21.15.49:6819/797242207","cluster_addr":"172.21.15.49:6821/797242207","heartbeat_back_addr":"172.21.15.49:6825/797242207","heartbeat_front_addr":"172.21.15.49:6823/797242207","state":["exists","up"]},{"osd":7,"uuid":"847d9bbb-7e28-4fe8-98b5-5acb1c2f2e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6817","nonce":1417643292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6819","nonce":1417643292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6823","nonce":1417643292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1417643292},{"type":"v1","addr":"172.21.15.79:6821","nonce":1417643292}]},"public_addr":"172.21.15.79:6817/1417643292","cluster_addr":"172.21.15.79:6819/1417643292","heartbeat_back_addr":"172.21.15.79:6823/1417643292","heartbeat_front_addr":"172.21.15.79:6821/1417643292","state":["exists","up"]},{"osd":8,"uuid":"14c55ac6-0e96-42a4-b974-178c87c2185c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6817","nonce":1465924719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6819","nonce":1465924719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6823","nonce":1465924719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1465924719},{"type":"v1","addr":"172.21.15.97:6821","nonce":1465924719}]},"public_addr":"172.21.15.97:6817/1465924719","cluster_addr":"172.21.15.97:6819/1465924719","heartbeat_back_addr":"172.21.15.97:6823/1465924719","heartbeat_front_addr":"172.21.15.97:6821/1465924719","state":["exists","up"]},{"osd":9,"uuid":"b0c07e94-8039-4e41-85b7-23344124687b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6827","nonce":845176131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6829","nonce":845176131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6833","nonce":845176131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":845176131},{"type":"v1","addr":"172.21.15.49:6831","nonce":845176131}]},"public_addr":"172.21.15.49:6827/845176131","cluster_addr":"172.21.15.49:6829/845176131","heartbeat_back_addr":"172.21.15.49:6833/845176131","heartbeat_front_addr":"172.21.15.49:6831/845176131","state":["exists","up"]},{"osd":10,"uuid":"67afc1c8-a2d3-4d0c-8b48-a90ad6fb3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6825","nonce":935518328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6827","nonce":935518328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6831","nonce":935518328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":935518328},{"type":"v1","addr":"172.21.15.79:6829","nonce":935518328}]},"public_addr":"172.21.15.79:6825/935518328","cluster_addr":"172.21.15.79:6827/935518328","heartbeat_back_addr":"172.21.15.79:6831/935518328","heartbeat_front_addr":"172.21.15.79:6829/935518328","state":["exists","up"]},{"osd":11,"uuid":"c2ad9939-51b9-481e-b454-8cfc7451065b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6825","nonce":1087893461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6827","nonce":1087893461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6831","nonce":1087893461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":1087893461},{"type":"v1","addr":"172.21.15.97:6829","nonce":1087893461}]},"public_addr":"172.21.15.97:6825/1087893461","cluster_addr":"172.21.15.97:6827/1087893461","heartbeat_back_addr":"172.21.15.97:6831/1087893461","heartbeat_front_addr":"172.21.15.97:6829/1087893461","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:16.085186+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:37.523233+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:44:58.760655+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:45:20.737876+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:45:41.148757+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:02.139676+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:23.295222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:46:43.460985+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:04.429151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:25.761472+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:47:46.048313+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T17:48:07.381968+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3795991287":"2023-07-21T17:42:52.746152+0000","172.21.15.49:0/3777764053":"2023-07-21T17:42:52.746152+0000","172.21.15.49:6801/752420140":"2023-07-21T17:42:52.746152+0000","172.21.15.49:0/2521746017":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/1021575989":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/3362614128":"2023-07-21T17:42:52.746152+0000","172.21.15.49:6801/3017374204":"2023-07-21T17:42:26.351231+0000","172.21.15.49:0/1674833015":"2023-07-21T17:42:26.351231+0000","172.21.15.49:6800/3017374204":"2023-07-21T17:42:26.351231+0000","172.21.15.49:6800/752420140":"2023-07-21T17:42:52.746152+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T17:48:52.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 -- 172.21.15.49:0/1514087207 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f19940a11d0 msgr2=0x7f19940a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:52.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f19940a11d0 0x7f19940a3690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1998005950 tx=0x7f199800b410 comp rx=0 tx=0).stop 2023-07-20T17:48:52.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 -- 172.21.15.49:0/1514087207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19a8103c40 msgr2=0x7f19a81969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:52.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19a8103c40 0x7f19a81969b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1990004730 tx=0x7f1990004760 comp rx=0 tx=0).stop 2023-07-20T17:48:52.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 -- 172.21.15.49:0/1514087207 shutdown_connections 2023-07-20T17:48:52.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f19940a11d0 0x7f19940a3690 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:52.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19a8106570 0x7f19a819da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:52.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19a8101310 0x7f19a8196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:52.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 --2- 172.21.15.49:0/1514087207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19a8103c40 0x7f19a81969b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:52.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 -- 172.21.15.49:0/1514087207 >> 172.21.15.49:0/1514087207 conn(0x7f19a80faca0 msgr2=0x7f19a8101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:52.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 -- 172.21.15.49:0/1514087207 shutdown_connections 2023-07-20T17:48:52.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.061+0000 7f19ae15d700 1 -- 172.21.15.49:0/1514087207 wait complete. 2023-07-20T17:48:52.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-20T17:48:52.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.215+0000 7f401b65a700 1 -- 172.21.15.49:0/1878089243 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4014101320 msgr2=0x7f4014103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:52.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.215+0000 7f401b65a700 1 --2- 172.21.15.49:0/1878089243 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4014101320 0x7f4014103710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4010009990 tx=0x7f4010009ca0 comp rx=0 tx=0).stop 2023-07-20T17:48:52.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.215+0000 7f401b65a700 1 -- 172.21.15.49:0/1878089243 shutdown_connections 2023-07-20T17:48:52.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.215+0000 7f401b65a700 1 --2- 172.21.15.49:0/1878089243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4014106580 0x7f4014108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:52.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.215+0000 7f401b65a700 1 --2- 172.21.15.49:0/1878089243 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4014101320 0x7f4014103710 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:52.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.215+0000 7f401b65a700 1 --2- 172.21.15.49:0/1878089243 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4014103c50 0x7f4014106040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:52.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.215+0000 7f401b65a700 1 -- 172.21.15.49:0/1878089243 >> 172.21.15.49:0/1878089243 conn(0x7f40140facf0 msgr2=0x7f40140fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:52.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.215+0000 7f401b65a700 1 -- 172.21.15.49:0/1878089243 shutdown_connections 2023-07-20T17:48:52.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.215+0000 7f401b65a700 1 -- 172.21.15.49:0/1878089243 wait complete. 2023-07-20T17:48:52.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f401b65a700 1 Processor -- start 2023-07-20T17:48:52.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f401b65a700 1 -- start start 2023-07-20T17:48:52.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f401b65a700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4014101320 0x7f4014196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:52.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f401b65a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4014103c50 0x7f4014196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:52.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f401b65a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4014106580 0x7f401419dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:52.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f401b65a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401419e090 con 0x7f4014106580 2023-07-20T17:48:52.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f401b65a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401419e1d0 con 0x7f4014103c50 2023-07-20T17:48:52.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f401b65a700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401419e360 con 0x7f4014101320 2023-07-20T17:48:52.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f4019bf7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4014106580 0x7f401419dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:52.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f4019bf7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4014106580 0x7f401419dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36504/0 (socket says 172.21.15.49:36504) 2023-07-20T17:48:52.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f4019bf7700 1 -- 172.21.15.49:0/73893031 learned_addr learned my addr 172.21.15.49:0/73893031 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:52.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.216+0000 7f4018bf5700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4014103c50 0x7f4014196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:52.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f40193f6700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4014101320 0x7f4014196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:52.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f4019bf7700 1 -- 172.21.15.49:0/73893031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4014101320 msgr2=0x7f4014196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:52.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f4019bf7700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4014101320 0x7f4014196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:52.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f4019bf7700 1 -- 172.21.15.49:0/73893031 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4014103c50 msgr2=0x7f4014196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:52.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f4019bf7700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4014103c50 0x7f4014196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:52.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f4019bf7700 1 -- 172.21.15.49:0/73893031 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40100095f0 con 0x7f4014106580 2023-07-20T17:48:52.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f40193f6700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4014101320 0x7f4014196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:48:52.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f4019bf7700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4014106580 0x7f401419dac0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f400c00ef30 tx=0x7f400c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:52.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f400a7fc700 1 -- 172.21.15.49:0/73893031 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c018070 con 0x7f4014106580 2023-07-20T17:48:52.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f400a7fc700 1 -- 172.21.15.49:0/73893031 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f400c00f040 con 0x7f4014106580 2023-07-20T17:48:52.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f401b65a700 1 -- 172.21.15.49:0/73893031 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401419e5e0 con 0x7f4014106580 2023-07-20T17:48:52.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f400a7fc700 1 -- 172.21.15.49:0/73893031 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00cdb0 con 0x7f4014106580 2023-07-20T17:48:52.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.217+0000 7f401b65a700 1 -- 172.21.15.49:0/73893031 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401419eaa0 con 0x7f4014106580 2023-07-20T17:48:52.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.218+0000 7f401b65a700 1 -- 172.21.15.49:0/73893031 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40140fc870 con 0x7f4014106580 2023-07-20T17:48:52.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.219+0000 7f400a7fc700 1 -- 172.21.15.49:0/73893031 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f400c003a00 con 0x7f4014106580 2023-07-20T17:48:52.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.220+0000 7f400a7fc700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f40000a1150 0x7f40000a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:52.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.220+0000 7f40193f6700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f40000a1150 0x7f40000a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:52.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.220+0000 7f400a7fc700 1 -- 172.21.15.49:0/73893031 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f400c0ccd60 con 0x7f4014106580 2023-07-20T17:48:52.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.221+0000 7f40193f6700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f40000a1150 0x7f40000a3610 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4010000c00 tx=0x7f4010005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:52.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.223+0000 7f400a7fc700 1 -- 172.21.15.49:0/73893031 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f400c096690 con 0x7f4014106580 2023-07-20T17:48:52.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:52 smithi049 ceph-mon[96530]: pgmap v249: 97 pgs: 4 creating+peering, 1 active+clean, 92 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:52.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:52 smithi049 ceph-mon[96530]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T17:48:52.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:52 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1933907616' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T17:48:52.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:52 smithi049 ceph-mon[96530]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T17:48:52.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:52 smithi049 ceph-mon[96530]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T17:48:52.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:52 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1111962888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:52.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:52 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1514087207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:52.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:52.500+0000 7f401b65a700 1 -- 172.21.15.49:0/73893031 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f401402c970 con 0x7f4014106580 2023-07-20T17:48:52.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:52 smithi097 ceph-mon[98398]: pgmap v249: 97 pgs: 4 creating+peering, 1 active+clean, 92 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:52.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:52 smithi097 ceph-mon[98398]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T17:48:52.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:52 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1933907616' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T17:48:52.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:52 smithi097 ceph-mon[98398]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T17:48:52.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:52 smithi097 ceph-mon[98398]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T17:48:52.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:52 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1111962888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:52.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:52 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1514087207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:52.567 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:52 smithi079 ceph-mon[98036]: pgmap v249: 97 pgs: 4 creating+peering, 1 active+clean, 92 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T17:48:52.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:52 smithi079 ceph-mon[98036]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T17:48:52.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:52 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1933907616' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T17:48:52.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:52 smithi079 ceph-mon[98036]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T17:48:52.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:52 smithi079 ceph-mon[98036]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T17:48:52.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:52 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1111962888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:52.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:52 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1514087207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T17:48:53.414 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.412+0000 7f400a7fc700 1 -- 172.21.15.49:0/73893031 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f400c099cb0 con 0x7f4014106580 2023-07-20T17:48:53.417 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.414+0000 7f401b65a700 1 -- 172.21.15.49:0/73893031 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f40000a1150 msgr2=0x7f40000a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:53.417 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.414+0000 7f401b65a700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f40000a1150 0x7f40000a3610 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4010000c00 tx=0x7f4010005fb0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.414+0000 7f401b65a700 1 -- 172.21.15.49:0/73893031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4014106580 msgr2=0x7f401419dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:53.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.414+0000 7f401b65a700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4014106580 0x7f401419dac0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f400c00ef30 tx=0x7f400c00c3a0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.415+0000 7f401b65a700 1 -- 172.21.15.49:0/73893031 shutdown_connections 2023-07-20T17:48:53.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.415+0000 7f401b65a700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f40000a1150 0x7f40000a3610 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.415+0000 7f401b65a700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4014101320 0x7f4014196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.415+0000 7f401b65a700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4014106580 0x7f401419dac0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.415+0000 7f401b65a700 1 --2- 172.21.15.49:0/73893031 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4014103c50 0x7f4014196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.415+0000 7f401b65a700 1 -- 172.21.15.49:0/73893031 >> 172.21.15.49:0/73893031 conn(0x7f40140facf0 msgr2=0x7f4014101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:53.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.415+0000 7f401b65a700 1 -- 172.21.15.49:0/73893031 shutdown_connections 2023-07-20T17:48:53.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.415+0000 7f401b65a700 1 -- 172.21.15.49:0/73893031 wait complete. 2023-07-20T17:48:53.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T17:48:53.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.79:6838/2994701526,v1:172.21.15.79:6839/2994701526] up:active 2023-07-20T17:48:53.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:53.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:53.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:53.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:53.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:53 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/73893031' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T17:48:53.434 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-20T17:48:53.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-07-20T17:48:53.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T17:48:53.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.518 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.79:6838/2994701526,v1:172.21.15.79:6839/2994701526] up:active 2023-07-20T17:48:53.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:53.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:53.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:53.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:53.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.520 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:53 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/73893031' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T17:48:53.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T17:48:53.568 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.569 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.79:6838/2994701526,v1:172.21.15.79:6839/2994701526] up:active 2023-07-20T17:48:53.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:53.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:53.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T17:48:53.570 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T17:48:53.571 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: from='mgr.14152 172.21.15.49:0/3062707762' entity='mgr.x' 2023-07-20T17:48:53.571 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:53 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/73893031' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T17:48:53.573 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.567+0000 7fd4ace20700 1 -- 172.21.15.49:0/2004628839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4a81023e0 msgr2=0x7fd4a8108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:53.573 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.567+0000 7fd4ace20700 1 --2- 172.21.15.49:0/2004628839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4a81023e0 0x7fd4a8108c90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000b0d0 tx=0x7fd4a000b3e0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.567+0000 7fd4ace20700 1 -- 172.21.15.49:0/2004628839 shutdown_connections 2023-07-20T17:48:53.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.567+0000 7fd4ace20700 1 --2- 172.21.15.49:0/2004628839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4a81023e0 0x7fd4a8108c90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.567+0000 7fd4ace20700 1 --2- 172.21.15.49:0/2004628839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4a8100850 0x7fd4a8100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.567+0000 7fd4ace20700 1 --2- 172.21.15.49:0/2004628839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4a8101a40 0x7fd4a8101ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.567+0000 7fd4ace20700 1 -- 172.21.15.49:0/2004628839 >> 172.21.15.49:0/2004628839 conn(0x7fd4a80fbde0 msgr2=0x7fd4a80fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:53.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.567+0000 7fd4ace20700 1 -- 172.21.15.49:0/2004628839 shutdown_connections 2023-07-20T17:48:53.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.568+0000 7fd4ace20700 1 -- 172.21.15.49:0/2004628839 wait complete. 2023-07-20T17:48:53.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.568+0000 7fd4ace20700 1 Processor -- start 2023-07-20T17:48:53.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.568+0000 7fd4ace20700 1 -- start start 2023-07-20T17:48:53.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.568+0000 7fd4ace20700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4a8100850 0x7fd4a81965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:53.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.568+0000 7fd4ace20700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4a8101a40 0x7fd4a8196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:53.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.568+0000 7fd4ace20700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4a81023e0 0x7fd4a819db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:53.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.568+0000 7fd4ace20700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a819e140 con 0x7fd4a8100850 2023-07-20T17:48:53.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.568+0000 7fd4ace20700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a819e280 con 0x7fd4a81023e0 2023-07-20T17:48:53.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.568+0000 7fd4ace20700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a819e410 con 0x7fd4a8101a40 2023-07-20T17:48:53.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4a8100850 0x7fd4a81965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:53.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4a8100850 0x7fd4a81965c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36512/0 (socket says 172.21.15.49:36512) 2023-07-20T17:48:53.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a659c700 1 -- 172.21.15.49:0/1089277643 learned_addr learned my addr 172.21.15.49:0/1089277643 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:53.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a5d9b700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4a8101a40 0x7fd4a8196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:53.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a6d9d700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4a81023e0 0x7fd4a819db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:53.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a659c700 1 -- 172.21.15.49:0/1089277643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4a8101a40 msgr2=0x7fd4a8196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:53.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a659c700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4a8101a40 0x7fd4a8196b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a659c700 1 -- 172.21.15.49:0/1089277643 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4a81023e0 msgr2=0x7fd4a819db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:53.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a659c700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4a81023e0 0x7fd4a819db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:53.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a659c700 1 -- 172.21.15.49:0/1089277643 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a0009d00 con 0x7fd4a8100850 2023-07-20T17:48:53.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a6d9d700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4a81023e0 0x7fd4a819db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T17:48:53.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4a659c700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4a8100850 0x7fd4a81965c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c970 tx=0x7fd49800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:53.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4977fe700 1 -- 172.21.15.49:0/1089277643 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498004220 con 0x7fd4a8100850 2023-07-20T17:48:53.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4977fe700 1 -- 172.21.15.49:0/1089277643 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd498007d60 con 0x7fd4a8100850 2023-07-20T17:48:53.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4ace20700 1 -- 172.21.15.49:0/1089277643 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a819e6f0 con 0x7fd4a8100850 2023-07-20T17:48:53.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4977fe700 1 -- 172.21.15.49:0/1089277643 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498003c90 con 0x7fd4a8100850 2023-07-20T17:48:53.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.569+0000 7fd4ace20700 1 -- 172.21.15.49:0/1089277643 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a819ec50 con 0x7fd4a8100850 2023-07-20T17:48:53.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.570+0000 7fd4ace20700 1 -- 172.21.15.49:0/1089277643 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a8107730 con 0x7fd4a8100850 2023-07-20T17:48:53.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.571+0000 7fd4977fe700 1 -- 172.21.15.49:0/1089277643 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd498004380 con 0x7fd4a8100850 2023-07-20T17:48:53.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.572+0000 7fd4977fe700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd4900a11d0 0x7fd4900a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:53.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.572+0000 7fd4a5d9b700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd4900a11d0 0x7fd4900a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:53.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.572+0000 7fd4977fe700 1 -- 172.21.15.49:0/1089277643 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd4980cc280 con 0x7fd4a8100850 2023-07-20T17:48:53.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.572+0000 7fd4a5d9b700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd4900a11d0 0x7fd4900a3690 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd49c005950 tx=0x7fd49c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:53.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.576+0000 7fd4977fe700 1 -- 172.21.15.49:0/1089277643 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd498095b20 con 0x7fd4a8100850 2023-07-20T17:48:53.825 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:53.823+0000 7fd4ace20700 1 -- 172.21.15.49:0/1089277643 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7fd4a802cbe0 con 0x7fd4a8100850 2023-07-20T17:48:54.417 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.414+0000 7fd4977fe700 1 -- 172.21.15.49:0/1089277643 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd498099140 con 0x7fd4a8100850 2023-07-20T17:48:54.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.417+0000 7fd4ace20700 1 -- 172.21.15.49:0/1089277643 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd4900a11d0 msgr2=0x7fd4900a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:54.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.417+0000 7fd4ace20700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd4900a11d0 0x7fd4900a3690 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd49c005950 tx=0x7fd49c00b410 comp rx=0 tx=0).stop 2023-07-20T17:48:54.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.417+0000 7fd4ace20700 1 -- 172.21.15.49:0/1089277643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4a8100850 msgr2=0x7fd4a81965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:54.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.417+0000 7fd4ace20700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4a8100850 0x7fd4a81965c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c970 tx=0x7fd49800cc80 comp rx=0 tx=0).stop 2023-07-20T17:48:54.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.418+0000 7fd4ace20700 1 -- 172.21.15.49:0/1089277643 shutdown_connections 2023-07-20T17:48:54.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.418+0000 7fd4ace20700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fd4900a11d0 0x7fd4900a3690 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:54.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.418+0000 7fd4ace20700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4a81023e0 0x7fd4a819db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:54.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.418+0000 7fd4ace20700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4a8100850 0x7fd4a81965c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:54.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.418+0000 7fd4ace20700 1 --2- 172.21.15.49:0/1089277643 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4a8101a40 0x7fd4a8196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:54.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.418+0000 7fd4ace20700 1 -- 172.21.15.49:0/1089277643 >> 172.21.15.49:0/1089277643 conn(0x7fd4a80fbde0 msgr2=0x7fd4a80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:54.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.418+0000 7fd4ace20700 1 -- 172.21.15.49:0/1089277643 shutdown_connections 2023-07-20T17:48:54.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.418+0000 7fd4ace20700 1 -- 172.21.15.49:0/1089277643 wait complete. 2023-07-20T17:48:54.438 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-20T17:48:54.573 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.570+0000 7f3957016700 1 -- 172.21.15.49:0/1645804900 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39501012d0 msgr2=0x7f3950107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:54.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.570+0000 7f3957016700 1 --2- 172.21.15.49:0/1645804900 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39501012d0 0x7f3950107b80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f394800b0d0 tx=0x7f394800b3e0 comp rx=0 tx=0).stop 2023-07-20T17:48:54.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.570+0000 7f3957016700 1 -- 172.21.15.49:0/1645804900 shutdown_connections 2023-07-20T17:48:54.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.570+0000 7f3957016700 1 --2- 172.21.15.49:0/1645804900 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39501012d0 0x7f3950107b80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:54.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.570+0000 7f3957016700 1 --2- 172.21.15.49:0/1645804900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500ff740 0x7f39500ffb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:54.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.570+0000 7f3957016700 1 --2- 172.21.15.49:0/1645804900 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3950100930 0x7f3950100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:54.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.570+0000 7f3957016700 1 -- 172.21.15.49:0/1645804900 >> 172.21.15.49:0/1645804900 conn(0x7f39500facf0 msgr2=0x7f39500fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:54.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.570+0000 7f3957016700 1 -- 172.21.15.49:0/1645804900 shutdown_connections 2023-07-20T17:48:54.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.570+0000 7f3957016700 1 -- 172.21.15.49:0/1645804900 wait complete. 2023-07-20T17:48:54.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f3957016700 1 Processor -- start 2023-07-20T17:48:54.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f3957016700 1 -- start start 2023-07-20T17:48:54.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f3957016700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500ff740 0x7f39501954a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:54.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f3957016700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3950100930 0x7f39501959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:54.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f3957016700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39501012d0 0x7f395019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:54.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f3957016700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395019d020 con 0x7f39501012d0 2023-07-20T17:48:54.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f3957016700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395019d160 con 0x7f39500ff740 2023-07-20T17:48:54.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f3957016700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395019d2f0 con 0x7f3950100930 2023-07-20T17:48:54.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f39555b3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39501012d0 0x7f395019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:54.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f39555b3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39501012d0 0x7f395019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36530/0 (socket says 172.21.15.49:36530) 2023-07-20T17:48:54.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f39555b3700 1 -- 172.21.15.49:0/908398252 learned_addr learned my addr 172.21.15.49:0/908398252 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:54.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f3954db2700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500ff740 0x7f39501954a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:54.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f394ffff700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3950100930 0x7f39501959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:54.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f39555b3700 1 -- 172.21.15.49:0/908398252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3950100930 msgr2=0x7f39501959e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:54.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f39555b3700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3950100930 0x7f39501959e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:54.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f39555b3700 1 -- 172.21.15.49:0/908398252 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500ff740 msgr2=0x7f39501954a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:54.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.571+0000 7f39555b3700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500ff740 0x7f39501954a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:54.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.572+0000 7f39555b3700 1 -- 172.21.15.49:0/908398252 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3948009d00 con 0x7f39501012d0 2023-07-20T17:48:54.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.572+0000 7f39555b3700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39501012d0 0x7f395019ca50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3948007a00 tx=0x7f39480094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:54.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.572+0000 7f394dffb700 1 -- 172.21.15.49:0/908398252 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394800e040 con 0x7f39501012d0 2023-07-20T17:48:54.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.572+0000 7f394dffb700 1 -- 172.21.15.49:0/908398252 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3948012dc0 con 0x7f39501012d0 2023-07-20T17:48:54.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.572+0000 7f3957016700 1 -- 172.21.15.49:0/908398252 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395019d570 con 0x7f39501012d0 2023-07-20T17:48:54.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.572+0000 7f394dffb700 1 -- 172.21.15.49:0/908398252 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394801b450 con 0x7f39501012d0 2023-07-20T17:48:54.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.572+0000 7f3957016700 1 -- 172.21.15.49:0/908398252 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395019d9f0 con 0x7f39501012d0 2023-07-20T17:48:54.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.573+0000 7f3957016700 1 -- 172.21.15.49:0/908398252 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3950106580 con 0x7f39501012d0 2023-07-20T17:48:54.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.575+0000 7f394dffb700 1 -- 172.21.15.49:0/908398252 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f394801d610 con 0x7f39501012d0 2023-07-20T17:48:54.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.575+0000 7f394dffb700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f39400a1150 0x7f39400a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:54.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.575+0000 7f394dffb700 1 -- 172.21.15.49:0/908398252 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f39480ced30 con 0x7f39501012d0 2023-07-20T17:48:54.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.575+0000 7f3954db2700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f39400a1150 0x7f39400a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:54.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.576+0000 7f3954db2700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f39400a1150 0x7f39400a3610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3944005950 tx=0x7f39440058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:54.582 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.579+0000 7f394dffb700 1 -- 172.21.15.49:0/908398252 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3948098660 con 0x7f39501012d0 2023-07-20T17:48:54.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:54 smithi049 ceph-mon[96530]: pgmap v252: 97 pgs: 46 active+clean, 4 creating+peering, 47 unknown; 578 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2023-07-20T17:48:54.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:54 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/73893031' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T17:48:54.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:54 smithi049 ceph-mon[96530]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:54.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:54 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1089277643' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T17:48:54.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:54 smithi097 ceph-mon[98398]: pgmap v252: 97 pgs: 46 active+clean, 4 creating+peering, 47 unknown; 578 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2023-07-20T17:48:54.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:54 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/73893031' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T17:48:54.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:54 smithi097 ceph-mon[98398]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:54.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:54 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1089277643' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T17:48:54.821 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:54 smithi079 ceph-mon[98036]: pgmap v252: 97 pgs: 46 active+clean, 4 creating+peering, 47 unknown; 578 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2023-07-20T17:48:54.821 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:54 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/73893031' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T17:48:54.822 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:54 smithi079 ceph-mon[98036]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:54.822 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:54 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1089277643' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T17:48:54.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:54.828+0000 7f3957016700 1 -- 172.21.15.49:0/908398252 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f395002cbc0 con 0x7f39501012d0 2023-07-20T17:48:55.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.424+0000 7f394dffb700 1 -- 172.21.15.49:0/908398252 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f394809bc80 con 0x7f39501012d0 2023-07-20T17:48:55.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 -- 172.21.15.49:0/908398252 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f39400a1150 msgr2=0x7f39400a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:55.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f39400a1150 0x7f39400a3610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3944005950 tx=0x7f39440058e0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 -- 172.21.15.49:0/908398252 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39501012d0 msgr2=0x7f395019ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:55.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39501012d0 0x7f395019ca50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3948007a00 tx=0x7f39480094f0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 -- 172.21.15.49:0/908398252 shutdown_connections 2023-07-20T17:48:55.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f39400a1150 0x7f39400a3610 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39501012d0 0x7f395019ca50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500ff740 0x7f39501954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 --2- 172.21.15.49:0/908398252 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3950100930 0x7f39501959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 -- 172.21.15.49:0/908398252 >> 172.21.15.49:0/908398252 conn(0x7f39500facf0 msgr2=0x7f39500fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:55.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 -- 172.21.15.49:0/908398252 shutdown_connections 2023-07-20T17:48:55.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.427+0000 7f3957016700 1 -- 172.21.15.49:0/908398252 wait complete. 2023-07-20T17:48:55.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-20T17:48:55.583 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.579+0000 7fab1ba0c700 1 -- 172.21.15.49:0/71501765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab141012d0 msgr2=0x7fab14107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:55.584 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.579+0000 7fab1ba0c700 1 --2- 172.21.15.49:0/71501765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab141012d0 0x7fab14107b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00b0d0 tx=0x7fab0c00b3e0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.584 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.579+0000 7fab1ba0c700 1 -- 172.21.15.49:0/71501765 shutdown_connections 2023-07-20T17:48:55.584 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.579+0000 7fab1ba0c700 1 --2- 172.21.15.49:0/71501765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab140ff740 0x7fab140ffb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.584 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.579+0000 7fab1ba0c700 1 --2- 172.21.15.49:0/71501765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab141012d0 0x7fab14107b80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.584 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.579+0000 7fab1ba0c700 1 --2- 172.21.15.49:0/71501765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab14100930 0x7fab14100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.584 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.579+0000 7fab1ba0c700 1 -- 172.21.15.49:0/71501765 >> 172.21.15.49:0/71501765 conn(0x7fab140facf0 msgr2=0x7fab140fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:55.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.579+0000 7fab1ba0c700 1 -- 172.21.15.49:0/71501765 shutdown_connections 2023-07-20T17:48:55.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.579+0000 7fab1ba0c700 1 -- 172.21.15.49:0/71501765 wait complete. 2023-07-20T17:48:55.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab1ba0c700 1 Processor -- start 2023-07-20T17:48:55.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab1ba0c700 1 -- start start 2023-07-20T17:48:55.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab1ba0c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab140ff740 0x7fab141954d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:55.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab1ba0c700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab14100930 0x7fab14195a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:55.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab1ba0c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab141012d0 0x7fab1419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:55.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab1ba0c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1419d050 con 0x7fab141012d0 2023-07-20T17:48:55.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab1ba0c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1419d190 con 0x7fab140ff740 2023-07-20T17:48:55.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab1ba0c700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1419d320 con 0x7fab14100930 2023-07-20T17:48:55.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab19fa9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab141012d0 0x7fab1419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:55.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab19fa9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab141012d0 0x7fab1419ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36544/0 (socket says 172.21.15.49:36544) 2023-07-20T17:48:55.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab19fa9700 1 -- 172.21.15.49:0/3536275539 learned_addr learned my addr 172.21.15.49:0/3536275539 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:55.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab197a8700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab140ff740 0x7fab141954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:55.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab18fa7700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab14100930 0x7fab14195a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:55.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab19fa9700 1 -- 172.21.15.49:0/3536275539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab14100930 msgr2=0x7fab14195a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:55.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab19fa9700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab14100930 0x7fab14195a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab19fa9700 1 -- 172.21.15.49:0/3536275539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab140ff740 msgr2=0x7fab141954d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:55.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab19fa9700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab140ff740 0x7fab141954d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:55.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.580+0000 7fab19fa9700 1 -- 172.21.15.49:0/3536275539 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c009d00 con 0x7fab141012d0 2023-07-20T17:48:55.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.581+0000 7fab19fa9700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab141012d0 0x7fab1419ca80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fab0c007a00 tx=0x7fab0c0094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:55.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.581+0000 7fab067fc700 1 -- 172.21.15.49:0/3536275539 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c00e040 con 0x7fab141012d0 2023-07-20T17:48:55.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.581+0000 7fab1ba0c700 1 -- 172.21.15.49:0/3536275539 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1419d5a0 con 0x7fab141012d0 2023-07-20T17:48:55.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.581+0000 7fab067fc700 1 -- 172.21.15.49:0/3536275539 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab0c003e10 con 0x7fab141012d0 2023-07-20T17:48:55.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.581+0000 7fab067fc700 1 -- 172.21.15.49:0/3536275539 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c01b9e0 con 0x7fab141012d0 2023-07-20T17:48:55.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.581+0000 7fab1ba0c700 1 -- 172.21.15.49:0/3536275539 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1419d970 con 0x7fab141012d0 2023-07-20T17:48:55.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.582+0000 7fab1ba0c700 1 -- 172.21.15.49:0/3536275539 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1410af50 con 0x7fab141012d0 2023-07-20T17:48:55.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.583+0000 7fab067fc700 1 -- 172.21.15.49:0/3536275539 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fab0c005630 con 0x7fab141012d0 2023-07-20T17:48:55.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.583+0000 7fab067fc700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab000a1150 0x7fab000a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:55.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.584+0000 7fab067fc700 1 -- 172.21.15.49:0/3536275539 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fab0c0ce350 con 0x7fab141012d0 2023-07-20T17:48:55.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.584+0000 7fab197a8700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab000a1150 0x7fab000a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:55.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.584+0000 7fab197a8700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab000a1150 0x7fab000a3610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fab14100790 tx=0x7fab1000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:55.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.587+0000 7fab067fc700 1 -- 172.21.15.49:0/3536275539 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fab0c097be0 con 0x7fab141012d0 2023-07-20T17:48:55.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:55 smithi049 ceph-mon[96530]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T17:48:55.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:55 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/1089277643' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T17:48:55.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:55 smithi049 ceph-mon[96530]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T17:48:55.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:55 smithi049 ceph-mon[96530]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:55.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:55 smithi049 ceph-mon[96530]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-20T17:48:55.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:55 smithi049 ceph-mon[96530]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-20T17:48:55.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:55 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/908398252' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T17:48:55.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:55 smithi097 ceph-mon[98398]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T17:48:55.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:55 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/1089277643' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T17:48:55.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:55 smithi097 ceph-mon[98398]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T17:48:55.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:55 smithi097 ceph-mon[98398]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:55.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:55 smithi097 ceph-mon[98398]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-20T17:48:55.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:55 smithi097 ceph-mon[98398]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-20T17:48:55.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:55 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/908398252' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T17:48:55.817 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:55 smithi079 ceph-mon[98036]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T17:48:55.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:55 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/1089277643' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T17:48:55.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:55 smithi079 ceph-mon[98036]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T17:48:55.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:55 smithi079 ceph-mon[98036]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T17:48:55.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:55 smithi079 ceph-mon[98036]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-20T17:48:55.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:55 smithi079 ceph-mon[98036]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-20T17:48:55.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:55 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/908398252' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T17:48:55.839 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:55.836+0000 7fab1ba0c700 1 -- 172.21.15.49:0/3536275539 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fab141065a0 con 0x7fab141012d0 2023-07-20T17:48:56.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.431+0000 7fab067fc700 1 -- 172.21.15.49:0/3536275539 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7fab0c09b200 con 0x7fab141012d0 2023-07-20T17:48:56.436 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 -- 172.21.15.49:0/3536275539 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab000a1150 msgr2=0x7fab000a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:56.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab000a1150 0x7fab000a3610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fab14100790 tx=0x7fab1000b410 comp rx=0 tx=0).stop 2023-07-20T17:48:56.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 -- 172.21.15.49:0/3536275539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab141012d0 msgr2=0x7fab1419ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:56.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab141012d0 0x7fab1419ca80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fab0c007a00 tx=0x7fab0c0094c0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 -- 172.21.15.49:0/3536275539 shutdown_connections 2023-07-20T17:48:56.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fab000a1150 0x7fab000a3610 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab141012d0 0x7fab1419ca80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab140ff740 0x7fab141954d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 --2- 172.21.15.49:0/3536275539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fab14100930 0x7fab14195a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 -- 172.21.15.49:0/3536275539 >> 172.21.15.49:0/3536275539 conn(0x7fab140facf0 msgr2=0x7fab140fce90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:56.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 -- 172.21.15.49:0/3536275539 shutdown_connections 2023-07-20T17:48:56.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.434+0000 7fab1ba0c700 1 -- 172.21.15.49:0/3536275539 wait complete. 2023-07-20T17:48:56.456 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2023-07-20T17:48:56.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2023-07-20T17:48:56.593 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.591+0000 7fea6e40e700 1 -- 172.21.15.49:0/2857708632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea68100320 msgr2=0x7fea68107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:56.594 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.591+0000 7fea6e40e700 1 --2- 172.21.15.49:0/2857708632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea68100320 0x7fea68107bc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fea6000b0a0 tx=0x7fea6000b3b0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.594 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.591+0000 7fea6e40e700 1 -- 172.21.15.49:0/2857708632 shutdown_connections 2023-07-20T17:48:56.594 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.591+0000 7fea6e40e700 1 --2- 172.21.15.49:0/2857708632 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea68100d10 0x7fea68101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.594 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.591+0000 7fea6e40e700 1 --2- 172.21.15.49:0/2857708632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea68100320 0x7fea68107bc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.594 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.591+0000 7fea6e40e700 1 --2- 172.21.15.49:0/2857708632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea680ff960 0x7fea680ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.594 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.591+0000 7fea6e40e700 1 -- 172.21.15.49:0/2857708632 >> 172.21.15.49:0/2857708632 conn(0x7fea680fa450 msgr2=0x7fea680fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:56.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.592+0000 7fea6e40e700 1 -- 172.21.15.49:0/2857708632 shutdown_connections 2023-07-20T17:48:56.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.592+0000 7fea6e40e700 1 -- 172.21.15.49:0/2857708632 wait complete. 2023-07-20T17:48:56.596 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.592+0000 7fea6e40e700 1 Processor -- start 2023-07-20T17:48:56.597 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.592+0000 7fea6e40e700 1 -- start start 2023-07-20T17:48:56.597 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6e40e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea680ff960 0x7fea68195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:56.597 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6e40e700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea68100320 0x7fea68195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:56.597 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6e40e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea68100d10 0x7fea6819cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:56.597 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6e40e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6819d0d0 con 0x7fea68100d10 2023-07-20T17:48:56.597 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6e40e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6819d210 con 0x7fea680ff960 2023-07-20T17:48:56.598 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6e40e700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6819d3a0 con 0x7fea68100320 2023-07-20T17:48:56.598 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6c9ab700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea68100d10 0x7fea6819cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:56.598 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6c9ab700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea68100d10 0x7fea6819cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36558/0 (socket says 172.21.15.49:36558) 2023-07-20T17:48:56.598 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6c9ab700 1 -- 172.21.15.49:0/2288861992 learned_addr learned my addr 172.21.15.49:0/2288861992 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:56.598 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea67fff700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea680ff960 0x7fea68195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:56.598 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea677fe700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea68100320 0x7fea68195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:56.598 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6c9ab700 1 -- 172.21.15.49:0/2288861992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea68100320 msgr2=0x7fea68195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:56.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6c9ab700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea68100320 0x7fea68195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6c9ab700 1 -- 172.21.15.49:0/2288861992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea680ff960 msgr2=0x7fea68195550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:56.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6c9ab700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea680ff960 0x7fea68195550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6c9ab700 1 -- 172.21.15.49:0/2288861992 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea60009d00 con 0x7fea68100d10 2023-07-20T17:48:56.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.593+0000 7fea6c9ab700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea68100d10 0x7fea6819cb00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fea6000d010 tx=0x7fea6000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:56.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.594+0000 7fea657fa700 1 -- 172.21.15.49:0/2288861992 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6000e040 con 0x7fea68100d10 2023-07-20T17:48:56.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.594+0000 7fea657fa700 1 -- 172.21.15.49:0/2288861992 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea6000bdb0 con 0x7fea68100d10 2023-07-20T17:48:56.599 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.594+0000 7fea6e40e700 1 -- 172.21.15.49:0/2288861992 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea680fe850 con 0x7fea68100d10 2023-07-20T17:48:56.600 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.594+0000 7fea657fa700 1 -- 172.21.15.49:0/2288861992 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea600052a0 con 0x7fea68100d10 2023-07-20T17:48:56.600 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.594+0000 7fea6e40e700 1 -- 172.21.15.49:0/2288861992 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea680fed60 con 0x7fea68100d10 2023-07-20T17:48:56.600 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.595+0000 7fea6e40e700 1 -- 172.21.15.49:0/2288861992 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6810afa0 con 0x7fea68100d10 2023-07-20T17:48:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.596+0000 7fea657fa700 1 -- 172.21.15.49:0/2288861992 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fea60005400 con 0x7fea68100d10 2023-07-20T17:48:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.597+0000 7fea657fa700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fea500a1150 0x7fea500a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.597+0000 7fea67fff700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fea500a1150 0x7fea500a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.597+0000 7fea657fa700 1 -- 172.21.15.49:0/2288861992 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fea600cf650 con 0x7fea68100d10 2023-07-20T17:48:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.597+0000 7fea67fff700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fea500a1150 0x7fea500a3610 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fea58009c60 tx=0x7fea58009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.600+0000 7fea657fa700 1 -- 172.21.15.49:0/2288861992 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fea60098f80 con 0x7fea68100d10 2023-07-20T17:48:56.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-mon[96530]: pgmap v253: 97 pgs: 83 active+clean, 1 creating+peering, 13 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2023-07-20T17:48:56.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.97:6838/2639990675,v1:172.21.15.97:6839/2639990675] up:active 2023-07-20T17:48:56.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/908398252' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T17:48:56.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-mon[96530]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T17:48:56.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-mon[96530]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-20T17:48:56.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-mon[96530]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-20T17:48:56.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-mon[96530]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-20T17:48:56.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3536275539' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T17:48:56.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:56 smithi097 ceph-mon[98398]: pgmap v253: 97 pgs: 83 active+clean, 1 creating+peering, 13 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2023-07-20T17:48:56.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:56 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.97:6838/2639990675,v1:172.21.15.97:6839/2639990675] up:active 2023-07-20T17:48:56.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:56 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/908398252' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T17:48:56.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:56 smithi097 ceph-mon[98398]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T17:48:56.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:56 smithi097 ceph-mon[98398]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-20T17:48:56.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:56 smithi097 ceph-mon[98398]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-20T17:48:56.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:56 smithi097 ceph-mon[98398]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-20T17:48:56.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:56 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3536275539' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T17:48:56.817 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:48:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-b[123332]: 2023-07-20T17:48:56.333+0000 7f831b06c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-20T17:48:56.818 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:48:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-b[123332]: 2023-07-20T17:48:56.333+0000 7f831b06c700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-07-20T17:48:56.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:56 smithi079 ceph-mon[98036]: pgmap v253: 97 pgs: 83 active+clean, 1 creating+peering, 13 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2023-07-20T17:48:56.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:56 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.97:6838/2639990675,v1:172.21.15.97:6839/2639990675] up:active 2023-07-20T17:48:56.818 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:56 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/908398252' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T17:48:56.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:56 smithi079 ceph-mon[98036]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T17:48:56.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:56 smithi079 ceph-mon[98036]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-20T17:48:56.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:56 smithi079 ceph-mon[98036]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-20T17:48:56.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:56 smithi079 ceph-mon[98036]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-20T17:48:56.819 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:56 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3536275539' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T17:48:56.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.834+0000 7fea6e40e700 1 -- 172.21.15.49:0/2288861992 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fea68060bb0 con 0x7fea500a1150 2023-07-20T17:48:56.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.861+0000 7fea657fa700 1 -- 172.21.15.49:0/2288861992 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fea68060bb0 con 0x7fea500a1150 2023-07-20T17:48:56.865 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.863+0000 7fea6e40e700 1 -- 172.21.15.49:0/2288861992 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fea500a1150 msgr2=0x7fea500a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:56.865 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.863+0000 7fea6e40e700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fea500a1150 0x7fea500a3610 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fea58009c60 tx=0x7fea58009260 comp rx=0 tx=0).stop 2023-07-20T17:48:56.866 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.863+0000 7fea6e40e700 1 -- 172.21.15.49:0/2288861992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea68100d10 msgr2=0x7fea6819cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:56.866 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.863+0000 7fea6e40e700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea68100d10 0x7fea6819cb00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fea6000d010 tx=0x7fea6000bc80 comp rx=0 tx=0).stop 2023-07-20T17:48:56.866 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.863+0000 7fea6e40e700 1 -- 172.21.15.49:0/2288861992 shutdown_connections 2023-07-20T17:48:56.866 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.863+0000 7fea6e40e700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7fea500a1150 0x7fea500a3610 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.866 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.863+0000 7fea6e40e700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea68100d10 0x7fea6819cb00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.866 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.863+0000 7fea6e40e700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea680ff960 0x7fea68195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.866 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.863+0000 7fea6e40e700 1 --2- 172.21.15.49:0/2288861992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fea68100320 0x7fea68195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:56.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.863+0000 7fea6e40e700 1 -- 172.21.15.49:0/2288861992 >> 172.21.15.49:0/2288861992 conn(0x7fea680fa450 msgr2=0x7fea680fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:56.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.863+0000 7fea6e40e700 1 -- 172.21.15.49:0/2288861992 shutdown_connections 2023-07-20T17:48:56.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:56.864+0000 7fea6e40e700 1 -- 172.21.15.49:0/2288861992 wait complete. 2023-07-20T17:48:56.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa '' 2023-07-20T17:48:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.009+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2997607085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c1012f0 msgr2=0x7f0d0c107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:57.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.009+0000 7f0d12fc4700 1 --2- 172.21.15.49:0/2997607085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c1012f0 0x7f0d0c107ba0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400b0d0 tx=0x7f0d0400b3e0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.009+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2997607085 shutdown_connections 2023-07-20T17:48:57.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.009+0000 7f0d12fc4700 1 --2- 172.21.15.49:0/2997607085 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d0c0ff760 0x7f0d0c0ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.009+0000 7f0d12fc4700 1 --2- 172.21.15.49:0/2997607085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c1012f0 0x7f0d0c107ba0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.009+0000 7f0d12fc4700 1 --2- 172.21.15.49:0/2997607085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d0c100950 0x7f0d0c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.009+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2997607085 >> 172.21.15.49:0/2997607085 conn(0x7f0d0c0facd0 msgr2=0x7f0d0c0fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:57.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2997607085 shutdown_connections 2023-07-20T17:48:57.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2997607085 wait complete. 2023-07-20T17:48:57.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d12fc4700 1 Processor -- start 2023-07-20T17:48:57.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d12fc4700 1 -- start start 2023-07-20T17:48:57.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d12fc4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c0ff760 0x7f0d0c10a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:57.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d12fc4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d0c100950 0x7f0d0c1086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:57.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d12fc4700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d0c1012f0 0x7f0d0c108bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:57.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d12fc4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c109390 con 0x7f0d0c0ff760 2023-07-20T17:48:57.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d12fc4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c109500 con 0x7f0d0c100950 2023-07-20T17:48:57.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d12fc4700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c1a0620 con 0x7f0d0c1012f0 2023-07-20T17:48:57.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d10d60700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c0ff760 0x7f0d0c10a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:57.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d10d60700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c0ff760 0x7f0d0c10a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36584/0 (socket says 172.21.15.49:36584) 2023-07-20T17:48:57.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.010+0000 7f0d10d60700 1 -- 172.21.15.49:0/2166216988 learned_addr learned my addr 172.21.15.49:0/2166216988 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:57.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d11561700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d0c1012f0 0x7f0d0c108bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:57.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d0bfff700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d0c100950 0x7f0d0c1086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:57.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d10d60700 1 -- 172.21.15.49:0/2166216988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d0c1012f0 msgr2=0x7f0d0c108bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:57.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d10d60700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d0c1012f0 0x7f0d0c108bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d10d60700 1 -- 172.21.15.49:0/2166216988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d0c100950 msgr2=0x7f0d0c1086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:57.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d10d60700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d0c100950 0x7f0d0c1086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d10d60700 1 -- 172.21.15.49:0/2166216988 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d04009d00 con 0x7f0d0c0ff760 2023-07-20T17:48:57.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d10d60700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c0ff760 0x7f0d0c10a070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000e970 tx=0x7f0d0000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:57.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d09ffb700 1 -- 172.21.15.49:0/2166216988 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d00009790 con 0x7f0d0c0ff760 2023-07-20T17:48:57.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d09ffb700 1 -- 172.21.15.49:0/2166216988 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d000044e0 con 0x7f0d0c0ff760 2023-07-20T17:48:57.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2166216988 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c1a0900 con 0x7f0d0c0ff760 2023-07-20T17:48:57.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d09ffb700 1 -- 172.21.15.49:0/2166216988 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d00003e30 con 0x7f0d0c0ff760 2023-07-20T17:48:57.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.011+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2166216988 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c1a0e60 con 0x7f0d0c0ff760 2023-07-20T17:48:57.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.012+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2166216988 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c10c120 con 0x7f0d0c0ff760 2023-07-20T17:48:57.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.013+0000 7f0d09ffb700 1 -- 172.21.15.49:0/2166216988 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0d00003f90 con 0x7f0d0c0ff760 2023-07-20T17:48:57.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.014+0000 7f0d09ffb700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0cfc0a11d0 0x7f0cfc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:57.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.014+0000 7f0d09ffb700 1 -- 172.21.15.49:0/2166216988 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0d00014070 con 0x7f0d0c0ff760 2023-07-20T17:48:57.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.014+0000 7f0d0bfff700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0cfc0a11d0 0x7f0cfc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:57.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.014+0000 7f0d0bfff700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0cfc0a11d0 0x7f0cfc0a3690 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8005fd0 tx=0x7f0cf8005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:57.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.018+0000 7f0d09ffb700 1 -- 172.21.15.49:0/2166216988 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0d00096300 con 0x7f0d0c0ff760 2023-07-20T17:48:57.240 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:48:56.839+0000 7f03af053700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T17:48:57.241 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:48:56.839+0000 7f03af053700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T17:48:57.241 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:48:56.839+0000 7f03af053700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T17:48:57.241 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:48:56.839+0000 7f03af053700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T17:48:57.241 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:48:56 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:48:56.839+0000 7f03af053700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T17:48:57.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.249+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2166216988 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d0c060bb0 con 0x7f0cfc0a11d0 2023-07-20T17:48:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.273+0000 7f0d09ffb700 1 -- 172.21.15.49:0/2166216988 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0d0c060bb0 con 0x7f0cfc0a11d0 2023-07-20T17:48:57.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2166216988 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0cfc0a11d0 msgr2=0x7f0cfc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0cfc0a11d0 0x7f0cfc0a3690 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8005fd0 tx=0x7f0cf8005f60 comp rx=0 tx=0).stop 2023-07-20T17:48:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2166216988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c0ff760 msgr2=0x7f0d0c10a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c0ff760 0x7f0d0c10a070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000e970 tx=0x7f0d0000ed30 comp rx=0 tx=0).stop 2023-07-20T17:48:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2166216988 shutdown_connections 2023-07-20T17:48:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f0cfc0a11d0 0x7f0cfc0a3690 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d0c1012f0 0x7f0d0c108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c0ff760 0x7f0d0c10a070 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 --2- 172.21.15.49:0/2166216988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d0c100950 0x7f0d0c1086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2166216988 >> 172.21.15.49:0/2166216988 conn(0x7f0d0c0facd0 msgr2=0x7f0d0c0fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:57.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2166216988 shutdown_connections 2023-07-20T17:48:57.279 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.275+0000 7f0d12fc4700 1 -- 172.21.15.49:0/2166216988 wait complete. 2023-07-20T17:48:57.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2023-07-20T17:48:57.426 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.424+0000 7f92720ec700 1 -- 172.21.15.49:0/3214053956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f926c100850 msgr2=0x7f926c100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:57.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.424+0000 7f92720ec700 1 --2- 172.21.15.49:0/3214053956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f926c100850 0x7f926c100c70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f925c0099c0 tx=0x7f925c009cd0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.425+0000 7f92720ec700 1 -- 172.21.15.49:0/3214053956 shutdown_connections 2023-07-20T17:48:57.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.425+0000 7f92720ec700 1 --2- 172.21.15.49:0/3214053956 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c1023e0 0x7f926c108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.425+0000 7f92720ec700 1 --2- 172.21.15.49:0/3214053956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f926c100850 0x7f926c100c70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.425+0000 7f92720ec700 1 --2- 172.21.15.49:0/3214053956 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f926c101a40 0x7f926c101ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.425+0000 7f92720ec700 1 -- 172.21.15.49:0/3214053956 >> 172.21.15.49:0/3214053956 conn(0x7f926c0fbdc0 msgr2=0x7f926c0fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T17:48:57.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.425+0000 7f92720ec700 1 -- 172.21.15.49:0/3214053956 shutdown_connections 2023-07-20T17:48:57.428 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.425+0000 7f92720ec700 1 -- 172.21.15.49:0/3214053956 wait complete. 2023-07-20T17:48:57.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.425+0000 7f92720ec700 1 Processor -- start 2023-07-20T17:48:57.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.425+0000 7f92720ec700 1 -- start start 2023-07-20T17:48:57.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f92720ec700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c100850 0x7f926c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:57.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f92720ec700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f926c101a40 0x7f926c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:57.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f92720ec700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f926c1023e0 0x7f926c19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:57.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f92720ec700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c1a02c0 con 0x7f926c1023e0 2023-07-20T17:48:57.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f92720ec700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c1a0400 con 0x7f926c100850 2023-07-20T17:48:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f92720ec700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c1a0590 con 0x7f926c101a40 2023-07-20T17:48:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f926c1023e0 0x7f926c19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c100850 0x7f926c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f926c1023e0 0x7f926c19fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36622/0 (socket says 172.21.15.49:36622) 2023-07-20T17:48:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926affd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f926c101a40 0x7f926c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c100850 0x7f926c198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.49:58440/0 (socket says 172.21.15.49:58440) 2023-07-20T17:48:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926bfff700 1 -- 172.21.15.49:0/3096605310 learned_addr learned my addr 172.21.15.49:0/3096605310 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-07-20T17:48:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926bfff700 1 -- 172.21.15.49:0/3096605310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f926c101a40 msgr2=0x7f926c198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926bfff700 1 --2- 172.21.15.49:0/3096605310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f926c101a40 0x7f926c198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926bfff700 1 -- 172.21.15.49:0/3096605310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c100850 msgr2=0x7f926c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T17:48:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926bfff700 1 --2- 172.21.15.49:0/3096605310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c100850 0x7f926c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T17:48:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926bfff700 1 -- 172.21.15.49:0/3096605310 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c0095f0 con 0x7f926c1023e0 2023-07-20T17:48:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f926bfff700 1 --2- 172.21.15.49:0/3096605310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f926c1023e0 0x7f926c19fcf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f926400b970 tx=0x7f926400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f9268ff9700 1 -- 172.21.15.49:0/3096605310 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926400d860 con 0x7f926c1023e0 2023-07-20T17:48:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f92720ec700 1 -- 172.21.15.49:0/3096605310 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c1a0870 con 0x7f926c1023e0 2023-07-20T17:48:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f9268ff9700 1 -- 172.21.15.49:0/3096605310 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f926400de80 con 0x7f926c1023e0 2023-07-20T17:48:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.426+0000 7f9268ff9700 1 -- 172.21.15.49:0/3096605310 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264017400 con 0x7f926c1023e0 2023-07-20T17:48:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.427+0000 7f92720ec700 1 -- 172.21.15.49:0/3096605310 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c1a0d30 con 0x7f926c1023e0 2023-07-20T17:48:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.428+0000 7f92720ec700 1 -- 172.21.15.49:0/3096605310 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c10c0e0 con 0x7f926c1023e0 2023-07-20T17:48:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.429+0000 7f9268ff9700 1 -- 172.21.15.49:0/3096605310 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f926400e630 con 0x7f926c1023e0 2023-07-20T17:48:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.429+0000 7f9268ff9700 1 --2- 172.21.15.49:0/3096605310 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f92540a1150 0x7f92540a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T17:48:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.430+0000 7f9268ff9700 1 -- 172.21.15.49:0/3096605310 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9264006e40 con 0x7f926c1023e0 2023-07-20T17:48:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.429+0000 7f926b7fe700 1 --2- 172.21.15.49:0/3096605310 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f92540a1150 0x7f92540a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T17:48:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.430+0000 7f926b7fe700 1 --2- 172.21.15.49:0/3096605310 >> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] conn(0x7f92540a1150 0x7f92540a3610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f925c009f50 tx=0x7f925c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T17:48:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.434+0000 7f9268ff9700 1 -- 172.21.15.49:0/3096605310 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9264095c80 con 0x7f926c1023e0 2023-07-20T17:48:57.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.669+0000 7f92720ec700 1 -- 172.21.15.49:0/3096605310 --> [v2:172.21.15.49:6800/971762269,v1:172.21.15.49:6801/971762269] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f926c1a13d0 con 0x7f92540a1150 2023-07-20T17:48:57.673 INFO:teuthology.orchestra.run.smithi049.stderr:2023-07-20T17:48:57.672+0000 7f9268ff9700 1 -- 172.21.15.49:0/3096605310 <== mgr.14152 v2:172.21.15.49:6800/971762269 1 ==== mgr_command_reply(tid 0: -2 subvolume 'sv_0' does not exist) v1 ==== 39+0+0 (secure 0 0 0) 0x7f926c1a13d0 con 0x7f92540a1150 2023-07-20T17:48:57.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.79:6836/3288313742,v1:172.21.15.79:6837/3288313742] up:active 2023-07-20T17:48:57.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: from='client.? 172.21.15.49:0/3536275539' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T17:48:57.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T17:48:57.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-20T17:48:57.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-20T17:48:57.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-20T17:48:57.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: mds.? [v2:172.21.15.97:6836/752441359,v1:172.21.15.97:6837/752441359] up:active 2023-07-20T17:48:57.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T17:48:57.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T17:48:57.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: Cluster is now healthy 2023-07-20T17:48:57.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2023-07-20T17:48:57.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2023-07-20T17:48:57.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-mon[96530]: daemon mds.a is now active in filesystem cephfs as rank 4 2023-07-20T17:48:57.676 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENOENT: subvolume 'sv_0' does not exist 2023-07-20T17:48:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-20T17:48:57.679 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' 2023-07-20T17:48:57.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T17:48:57.707 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T17:48:57.740 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.79:6836/3288313742,v1:172.21.15.79:6837/3288313742] up:active 2023-07-20T17:48:57.740 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: from='client.? 172.21.15.49:0/3536275539' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T17:48:57.740 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T17:48:57.740 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-20T17:48:57.741 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-20T17:48:57.741 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-20T17:48:57.741 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: mds.? [v2:172.21.15.97:6836/752441359,v1:172.21.15.97:6837/752441359] up:active 2023-07-20T17:48:57.741 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T17:48:57.741 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T17:48:57.742 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: Cluster is now healthy 2023-07-20T17:48:57.742 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2023-07-20T17:48:57.742 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2023-07-20T17:48:57.742 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:57 smithi079 ceph-mon[98036]: daemon mds.a is now active in filesystem cephfs as rank 4 2023-07-20T17:48:57.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T17:48:57.768 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.79:6836/3288313742,v1:172.21.15.79:6837/3288313742] up:active 2023-07-20T17:48:57.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: from='client.? 172.21.15.49:0/3536275539' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T17:48:57.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T17:48:57.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-20T17:48:57.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-20T17:48:57.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-20T17:48:57.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: mds.? [v2:172.21.15.97:6836/752441359,v1:172.21.15.97:6837/752441359] up:active 2023-07-20T17:48:57.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T17:48:57.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T17:48:57.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: Cluster is now healthy 2023-07-20T17:48:57.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2023-07-20T17:48:57.770 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2023-07-20T17:48:57.771 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:48:57 smithi097 ceph-mon[98398]: daemon mds.a is now active in filesystem cephfs as rank 4 2023-07-20T17:48:57.783 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-20T17:48:57.784 DEBUG:teuthology.orchestra.run.smithi049:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T17:48:57.801 DEBUG:teuthology.orchestra.run.smithi079:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T17:48:57.822 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T17:48:57.844 INFO:tasks.cephadm:Stopping all daemons... 2023-07-20T17:48:57.845 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-20T17:48:57.846 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.a 2023-07-20T17:48:57.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:57 smithi049 systemd[1]: Stopping Ceph mon.a for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:57.991 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:48:57 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:48:57.672+0000 7f03af053700 -1 mgr.server reply reply (2) No such file or directory subvolume 'sv_0' does not exist 2023-07-20T17:48:58.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:58 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-a[96506]: 2023-07-20T17:48:58.184+0000 7f4a6fa5e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:48:58.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:58 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-a[96506]: 2023-07-20T17:48:58.184+0000 7f4a6fa5e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-20T17:48:58.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Jul 20 17:48:58 smithi049 bash[133109]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-a 2023-07-20T17:48:58.881 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.a.service' 2023-07-20T17:48:58.932 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:48:58.933 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-20T17:48:58.933 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-20T17:48:58.934 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.b 2023-07-20T17:48:59.319 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:58 smithi079 systemd[1]: Stopping Ceph mon.b for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:48:59.636 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-b[98012]: 2023-07-20T17:48:59.364+0000 7fb659926700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:48:59.637 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-b[98012]: 2023-07-20T17:48:59.364+0000 7fb659926700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-20T17:49:00.067 INFO:journalctl@ceph.mon.b.smithi079.stdout:Jul 20 17:48:59 smithi079 bash[124562]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-b 2023-07-20T17:49:00.203 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.b.service' 2023-07-20T17:49:00.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:00.256 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-20T17:49:00.257 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-20T17:49:00.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.c 2023-07-20T17:49:00.686 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:49:00 smithi097 systemd[1]: Stopping Ceph mon.c for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:01.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:49:00 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-c[98361]: 2023-07-20T17:49:00.681+0000 7f293522c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:49:01.018 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:49:00 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-c[98361]: 2023-07-20T17:49:00.681+0000 7f293522c700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-20T17:49:01.449 INFO:journalctl@ceph.mon.c.smithi097.stdout:Jul 20 17:49:01 smithi097 bash[125945]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mon-c 2023-07-20T17:49:01.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mon.c.service' 2023-07-20T17:49:01.513 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:01.514 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-20T17:49:01.514 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-20T17:49:01.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T17:49:01.817 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:49:01 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-b[123332]: 2023-07-20T17:49:01.333+0000 7f831b06c700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-07-20T17:49:01.878 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:49:01 smithi049 systemd[1]: Stopping Ceph mgr.x for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:02.150 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:49:01 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x[96777]: 2023-07-20T17:49:01.874+0000 7f0440cc0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:49:02.406 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Jul 20 17:49:02 smithi049 bash[133626]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-x 2023-07-20T17:49:02.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.x.service' 2023-07-20T17:49:02.454 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:02.455 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-20T17:49:02.456 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-20T17:49:02.457 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.y 2023-07-20T17:49:02.818 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:49:02 smithi079 systemd[1]: Stopping Ceph mgr.y for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:03.168 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:49:03 smithi079 bash[124967]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-y 2023-07-20T17:49:03.169 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Jul 20 17:49:03 smithi079 systemd[1]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-20T17:49:03.353 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.y.service' 2023-07-20T17:49:03.392 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:03.392 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-20T17:49:03.393 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-20T17:49:03.393 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.z 2023-07-20T17:49:03.768 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:49:03 smithi097 systemd[1]: Stopping Ceph mgr.z for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:04.101 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:49:04 smithi097 bash[126135]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mgr-z 2023-07-20T17:49:04.284 INFO:journalctl@ceph.mgr.z.smithi097.stdout:Jul 20 17:49:04 smithi097 systemd[1]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-20T17:49:04.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mgr.z.service' 2023-07-20T17:49:04.325 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:04.325 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-20T17:49:04.326 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-20T17:49:04.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.0 2023-07-20T17:49:04.693 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:49:04 smithi049 systemd[1]: Stopping Ceph osd.0 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:04.990 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:49:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-0[109572]: 2023-07-20T17:49:04.690+0000 7faaa3648700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:49:04.990 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:49:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-0[109572]: 2023-07-20T17:49:04.690+0000 7faaa3648700 -1 osd.0 82 *** Got signal Terminated *** 2023-07-20T17:49:04.990 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:49:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-0[109572]: 2023-07-20T17:49:04.690+0000 7faaa3648700 -1 osd.0 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:49:10.740 INFO:journalctl@ceph.osd.0.smithi049.stdout:Jul 20 17:49:10 smithi049 bash[133769]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-0 2023-07-20T17:49:11.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.0.service' 2023-07-20T17:49:11.842 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:11.843 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-20T17:49:11.844 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-20T17:49:11.844 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.1 2023-07-20T17:49:12.192 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:49:11 smithi079 systemd[1]: Stopping Ceph osd.1 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:12.568 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:49:12 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-1[103103]: 2023-07-20T17:49:12.189+0000 7f0a6f087700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:49:12.568 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:49:12 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-1[103103]: 2023-07-20T17:49:12.189+0000 7f0a6f087700 -1 osd.1 82 *** Got signal Terminated *** 2023-07-20T17:49:12.568 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:49:12 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-1[103103]: 2023-07-20T17:49:12.189+0000 7f0a6f087700 -1 osd.1 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:49:18.068 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jul 20 17:49:17 smithi079 bash[125114]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-1 2023-07-20T17:49:19.044 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.1.service' 2023-07-20T17:49:19.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:19.084 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-20T17:49:19.085 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-20T17:49:19.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.2 2023-07-20T17:49:19.465 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:49:19 smithi097 systemd[1]: Stopping Ceph osd.2 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:19.768 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:49:19 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-2[104225]: 2023-07-20T17:49:19.461+0000 7f6347c50700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:49:19.768 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:49:19 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-2[104225]: 2023-07-20T17:49:19.461+0000 7f6347c50700 -1 osd.2 82 *** Got signal Terminated *** 2023-07-20T17:49:19.768 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:49:19 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-2[104225]: 2023-07-20T17:49:19.461+0000 7f6347c50700 -1 osd.2 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:49:25.518 INFO:journalctl@ceph.osd.2.smithi097.stdout:Jul 20 17:49:25 smithi097 bash[126284]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-2 2023-07-20T17:49:26.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.2.service' 2023-07-20T17:49:26.746 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:26.747 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-20T17:49:26.747 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-20T17:49:26.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.3 2023-07-20T17:49:27.114 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jul 20 17:49:26 smithi049 systemd[1]: Stopping Ceph osd.3 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:27.490 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jul 20 17:49:27 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-3[114991]: 2023-07-20T17:49:27.111+0000 7fdcaf9dc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:49:27.490 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jul 20 17:49:27 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-3[114991]: 2023-07-20T17:49:27.111+0000 7fdcaf9dc700 -1 osd.3 82 *** Got signal Terminated *** 2023-07-20T17:49:27.491 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jul 20 17:49:27 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-3[114991]: 2023-07-20T17:49:27.111+0000 7fdcaf9dc700 -1 osd.3 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:49:31.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:31.331+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:31.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:31 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:31.313+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:32.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:32 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:32.317+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:32.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:32 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:32.317+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:32.818 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:32 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:32.365+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:33.241 INFO:journalctl@ceph.osd.3.smithi049.stdout:Jul 20 17:49:32 smithi049 bash[134043]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-3 2023-07-20T17:49:33.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:33 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:33.232+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:33.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:33 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:33.175+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:33.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:33 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:33.361+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:33.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:33 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:33.298+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:33.818 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:33 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:33.343+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:34.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:33 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:33.722+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:34.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.3.service' 2023-07-20T17:49:34.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:34.207 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-20T17:49:34.208 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-20T17:49:34.208 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.4 2023-07-20T17:49:34.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:34 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:34.260+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:34.491 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:34 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:34.406+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:34.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:34 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:34.239+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:34.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:34 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:34.199+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:34.553 INFO:journalctl@ceph.osd.4.smithi079.stdout:Jul 20 17:49:34 smithi079 systemd[1]: Stopping Ceph osd.4 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:34.553 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:34 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:34.333+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:34.818 INFO:journalctl@ceph.osd.4.smithi079.stdout:Jul 20 17:49:34 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-4[108622]: 2023-07-20T17:49:34.550+0000 7fde49e95700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:49:34.818 INFO:journalctl@ceph.osd.4.smithi079.stdout:Jul 20 17:49:34 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-4[108622]: 2023-07-20T17:49:34.550+0000 7fde49e95700 -1 osd.4 82 *** Got signal Terminated *** 2023-07-20T17:49:34.818 INFO:journalctl@ceph.osd.4.smithi079.stdout:Jul 20 17:49:34 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-4[108622]: 2023-07-20T17:49:34.550+0000 7fde49e95700 -1 osd.4 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:49:35.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:34 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:34.727+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:35.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:35 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:35.208+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:35.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:35 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:35.187+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:35.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:35 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:35.363+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:35.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:35 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:35.261+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:35.818 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:35 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:35.340+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:36.020 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:35 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:35.712+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:36.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:36 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:36.251+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:36.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:36 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:36.201+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:36.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:36 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:36.373+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:36.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:36 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:36.289+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:36.818 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:36 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:36.331+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:37.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:36 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:36.689+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:37.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:37 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:37.263+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:37.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:37 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:37.221+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:37.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:37 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:37.364+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:37.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:37 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:37.335+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:37.818 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:37 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:37.339+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:38.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:37 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:37.652+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:38.654 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:38 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:38.267+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:38.654 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:38 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:38.269+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:38.654 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:38 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:38.269+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:38.740 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:38 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:38.337+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:38.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:38 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:38.378+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:38.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:38 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:38.378+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:38.773 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:38 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:38.364+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:39.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:38 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:38.652+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:39.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:38 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:38.652+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.970349+0000 front 2023-07-20T17:49:16.970315+0000 (oldest deadline 2023-07-20T17:49:38.070453+0000) 2023-07-20T17:49:39.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:38 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:38.771+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:39.568 INFO:journalctl@ceph.osd.4.smithi079.stdout:Jul 20 17:49:39 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-4[108622]: 2023-07-20T17:49:39.266+0000 7fde44488700 -1 osd.4 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:13.624509+0000 front 2023-07-20T17:49:13.624535+0000 (oldest deadline 2023-07-20T17:49:38.324661+0000) 2023-07-20T17:49:39.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:39 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:39.328+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:39.635 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:39 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:39.272+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:39.636 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:39 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:39.308+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:39.636 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:39 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:39.308+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:39.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:39 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:39.347+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:39.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:39 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:39.347+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:39.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:39 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:39.297+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:39.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:39 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:39.297+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:40.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:39 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:39.633+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:40.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:39 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:39.633+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.970349+0000 front 2023-07-20T17:49:16.970315+0000 (oldest deadline 2023-07-20T17:49:38.070453+0000) 2023-07-20T17:49:40.058 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:39 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:39.802+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:40.318 INFO:journalctl@ceph.osd.4.smithi079.stdout:Jul 20 17:49:40 smithi079 bash[125391]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-4 2023-07-20T17:49:40.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:40 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:40.288+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:40.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:40 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:40.224+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:40.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:40 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:40.316+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:40.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:40 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:40.316+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:40.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:40 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:40.377+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:40.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:40 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:40.377+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:40.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:40 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:40.294+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:40.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:40 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:40.294+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:41.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:40 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:40.664+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:41.019 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:40 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:40.664+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.970349+0000 front 2023-07-20T17:49:16.970315+0000 (oldest deadline 2023-07-20T17:49:38.070453+0000) 2023-07-20T17:49:41.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:40 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:40.776+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:41.494 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:41 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:41.338+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:41.495 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:41 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:41.338+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:41.501 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.4.service' 2023-07-20T17:49:41.541 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:41.542 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-20T17:49:41.542 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-20T17:49:41.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.5 2023-07-20T17:49:41.572 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:41 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:41.271+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:41.573 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:41 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:41.271+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:41.573 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:41 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:41.329+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:41.573 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:41 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:41.329+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:41.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:41 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:41.415+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:41.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:41 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:41.415+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:41.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:41 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:41.256+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:41.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:41 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:41.256+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:41.818 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:41 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:41.740+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:41.926 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:41 smithi097 systemd[1]: Stopping Ceph osd.5 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:41.927 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:41 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:41.699+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:41.927 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:41 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:41.699+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.970349+0000 front 2023-07-20T17:49:16.970315+0000 (oldest deadline 2023-07-20T17:49:38.070453+0000) 2023-07-20T17:49:41.927 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:41 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:41.924+0000 7f484eb8f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:49:41.927 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:41 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:41.924+0000 7f484eb8f700 -1 osd.5 82 *** Got signal Terminated *** 2023-07-20T17:49:42.268 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:41 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:41.924+0000 7f484eb8f700 -1 osd.5 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:49:42.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:42 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:42.418+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:42.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:42 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:42.418+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:42.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:42 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:42.302+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:42.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:42 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:42.302+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:42.742 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:42 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:42.310+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:42.742 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:42 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:42.310+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:42.742 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:42 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:42.360+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:42.742 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:42 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:42.360+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:42.754 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:42 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:42.323+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:42.754 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:42 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:42.323+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:43.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:42 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:42.740+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:43.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:42 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:42.740+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.970349+0000 front 2023-07-20T17:49:16.970315+0000 (oldest deadline 2023-07-20T17:49:38.070453+0000) 2023-07-20T17:49:43.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:42 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:42.752+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:43.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:43 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:43.292+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:43.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:43 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:43.292+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:43.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:43 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:43.401+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:43.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:43 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:43.401+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:43.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:43 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:43.266+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:43.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:43 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:43.266+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:43.754 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:43 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:43.310+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:43.754 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:43 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:43.310+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:43.754 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:43 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:43.315+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:43.754 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:43 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:43.315+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:44.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:43 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:43.751+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:44.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:43 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:43.751+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.970349+0000 front 2023-07-20T17:49:16.970315+0000 (oldest deadline 2023-07-20T17:49:38.070453+0000) 2023-07-20T17:49:44.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:43 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:43.763+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:44.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:44 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:44.259+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:44.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:44 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:44.259+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:44.740 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:44 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:44.307+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:44.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:44 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:44.307+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:44.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:44 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:44.389+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:44.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:44 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:44.389+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:44.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:44 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:44.305+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:44.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:44 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:44.305+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:44.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:44 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:44.334+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:44.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:44 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:44.334+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:45.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:44 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:44.741+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:45.268 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:44 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:44.777+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:45.269 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:44 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:44.777+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.970349+0000 front 2023-07-20T17:49:16.970315+0000 (oldest deadline 2023-07-20T17:49:38.070453+0000) 2023-07-20T17:49:45.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:45 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:45.261+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:45.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:45 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:45.261+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:45.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:45.363+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:45.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:45.363+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:45.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:45.307+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:45.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:45.307+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:45.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:45.307+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:45.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:45 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:45.330+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:45.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:45 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:45.330+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:45.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:45 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:45.330+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:45.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:45 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:45.339+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:45.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:45 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:45.339+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:45.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:45 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:45.339+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:46.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:45 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:45.710+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:46.268 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:45 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:45.802+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:46.268 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:45 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:45.802+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.970349+0000 front 2023-07-20T17:49:16.970315+0000 (oldest deadline 2023-07-20T17:49:38.070453+0000) 2023-07-20T17:49:46.268 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:45 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:45.802+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:20.971316+0000 front 2023-07-20T17:49:20.971341+0000 (oldest deadline 2023-07-20T17:49:45.071496+0000) 2023-07-20T17:49:46.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:46 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:46.269+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:46.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:46 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:46.269+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:46.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:46 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:46.400+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:46.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:46 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:46.400+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:46.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:46 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:46.287+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:46.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:46 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:46.287+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:46.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:46 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:46.287+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:46.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:46 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:46.286+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:46.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:46 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:46.286+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:46.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:46 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:46.286+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:46.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:46 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:46.379+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:46.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:46 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:46.379+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:46.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:46 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:46.379+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:47.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:46 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:46.729+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:47.268 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:46 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:46.796+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.769363+0000 front 2023-07-20T17:49:08.769155+0000 (oldest deadline 2023-07-20T17:49:32.869292+0000) 2023-07-20T17:49:47.269 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:46 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:46.796+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.970349+0000 front 2023-07-20T17:49:16.970315+0000 (oldest deadline 2023-07-20T17:49:38.070453+0000) 2023-07-20T17:49:47.269 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:46 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5[109678]: 2023-07-20T17:49:46.796+0000 7f4849182700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:20.971316+0000 front 2023-07-20T17:49:20.971341+0000 (oldest deadline 2023-07-20T17:49:45.071496+0000) 2023-07-20T17:49:47.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:47 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:47.260+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:47.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:47 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:47.260+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:47.729 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:47 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:47.339+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:47.729 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:47 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:47.339+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:47.729 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:47 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:47.339+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:47.729 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:47 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:47.292+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:47.730 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:47 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:47.292+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:47.730 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:47 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:47.292+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:47.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:47 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:47.413+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:47.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:47 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:47.413+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:47.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:47 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:47.413+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.216166+0000 front 2023-07-20T17:49:23.216106+0000 (oldest deadline 2023-07-20T17:49:46.716096+0000) 2023-07-20T17:49:47.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:47 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:47.303+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:47.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:47 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:47.303+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:47.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:47 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:47.303+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:48.018 INFO:journalctl@ceph.osd.5.smithi097.stdout:Jul 20 17:49:47 smithi097 bash[126555]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-5 2023-07-20T17:49:48.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:47 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:47.711+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:48.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:47 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:47.711+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:48.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:48 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:48.273+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:48.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:48 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:48.273+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:48.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:48 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:48.382+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:48.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:48 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:48.382+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:48.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:48 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:48.382+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.216166+0000 front 2023-07-20T17:49:23.216106+0000 (oldest deadline 2023-07-20T17:49:46.716096+0000) 2023-07-20T17:49:48.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:48 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:48.265+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:48.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:48 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:48.265+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:48.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:48 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:48.265+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:48.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:48 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:48.341+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:48.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:48 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:48.341+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:48.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:48 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:48.341+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:48.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:48 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:48.317+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:48.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:48 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:48.317+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:48.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:48 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:48.317+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:49.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:48 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:48.668+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:49.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:48 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:48.668+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:49.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.5.service' 2023-07-20T17:49:49.373 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:49.374 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-20T17:49:49.374 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-20T17:49:49.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.6 2023-07-20T17:49:49.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:49.234+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:49.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:49.234+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:49.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:49.234+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:49.492 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:49 smithi049 systemd[1]: Stopping Ceph osd.6 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:49.493 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:49.415+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:49.493 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:49.415+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:49.493 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:49.415+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.216166+0000 front 2023-07-20T17:49:23.216106+0000 (oldest deadline 2023-07-20T17:49:46.716096+0000) 2023-07-20T17:49:49.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:49 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:49.225+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:49.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:49 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:49.225+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:49.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:49 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:49.359+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:49.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:49 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:49.359+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:49.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:49 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:49.359+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:49.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:49 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:49.338+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:49.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:49 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:49.338+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:49.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:49 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:49.338+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:50.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:49 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:49.665+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:50.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:49 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:49.665+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:50.241 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:49.740+0000 7f2f3fe9a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:49:50.241 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:49.740+0000 7f2f3fe9a700 -1 osd.6 82 *** Got signal Terminated *** 2023-07-20T17:49:50.241 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:49 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:49.740+0000 7f2f3fe9a700 -1 osd.6 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:49:50.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:50 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:50.258+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:50.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:50 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:50.258+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:50.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:50 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:50.258+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:49:50.740 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:50 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:50.465+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:50.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:50 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:50.465+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:50.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:50 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:50.465+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.216166+0000 front 2023-07-20T17:49:23.216106+0000 (oldest deadline 2023-07-20T17:49:46.716096+0000) 2023-07-20T17:49:50.742 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:50 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:50.273+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:50.742 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:50 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:50.273+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:50.742 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:50 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:50.273+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:50.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:50 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:50.362+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:50.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:50 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:50.362+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:50.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:50 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:50.362+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:50.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:50 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:50.291+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:50.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:50 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:50.291+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:50.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:50 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:50.291+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:51.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:50 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:50.684+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:51.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:50 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:50.684+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:51.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:51 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:51.216+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:51.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:51 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:51.216+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:51.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:51 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:51.216+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:49:51.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:51 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:51.430+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:51.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:51 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:51.430+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:51.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:51 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:51.430+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.216166+0000 front 2023-07-20T17:49:23.216106+0000 (oldest deadline 2023-07-20T17:49:46.716096+0000) 2023-07-20T17:49:51.741 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:51 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:51.250+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:51.742 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:51 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:51.250+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:51.742 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:51 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:51.250+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:51.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:51 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:51.361+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:51.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:51 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:51.361+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:51.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:51 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:51.361+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:51.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:51 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:51.308+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:51.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:51 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:51.308+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:51.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:51 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:51.308+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:52.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:51 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:51.732+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:52.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:51 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:51.732+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:52.490 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:52 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:52.429+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:52.491 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:52 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:52.429+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:52.491 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:52 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:52.429+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.216166+0000 front 2023-07-20T17:49:23.216106+0000 (oldest deadline 2023-07-20T17:49:46.716096+0000) 2023-07-20T17:49:52.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:52 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:52.219+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:52.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:52 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:52.219+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:52.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:52 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:52.219+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:52.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:52 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:52.211+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:52.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:52 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:52.211+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:52.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:52 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:52.211+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:49:52.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:52 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:52.327+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:52.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:52 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:52.327+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:52.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:52 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:52.327+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:52.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:52 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:52.291+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:52.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:52 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:52.291+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:52.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:52 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:52.291+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:53.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:52 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:52.698+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:53.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:52 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:52.698+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:53.477 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:53 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:53.175+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:53.477 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:53 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:53.175+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:53.477 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:53 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:53.175+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:53.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:53 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:53.179+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:53.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:53 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:53.179+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:53.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:53 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:53.179+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:49:53.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:53 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:53.473+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:53.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:53 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:53.473+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:53.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:53 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:53.473+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.216166+0000 front 2023-07-20T17:49:23.216106+0000 (oldest deadline 2023-07-20T17:49:46.716096+0000) 2023-07-20T17:49:53.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:53 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:53.473+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.816989+0000 front 2023-07-20T17:49:30.817016+0000 (oldest deadline 2023-07-20T17:49:52.517188+0000) 2023-07-20T17:49:53.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:53.270+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:53.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:53.270+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:53.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:53.270+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:53.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:53.314+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:53.770 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:53.314+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:53.770 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:53.314+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:53.770 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:53 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:53.314+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:49:54.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:53 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:53.683+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:54.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:53 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:53.683+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:54.445 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:54 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:54.130+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:54.445 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:54 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:54.130+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:54.445 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:54 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:54.130+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:54.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:54 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:54.148+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:54.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:54 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:54.148+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:54.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:54 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:54.148+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:49:54.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:54 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:54.442+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.013715+0000 front 2023-07-20T17:49:08.013754+0000 (oldest deadline 2023-07-20T17:49:30.913967+0000) 2023-07-20T17:49:54.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:54 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:54.442+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.615156+0000 front 2023-07-20T17:49:12.615182+0000 (oldest deadline 2023-07-20T17:49:37.915093+0000) 2023-07-20T17:49:54.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:54 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:54.442+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.216166+0000 front 2023-07-20T17:49:23.216106+0000 (oldest deadline 2023-07-20T17:49:46.716096+0000) 2023-07-20T17:49:54.741 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:54 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6[120455]: 2023-07-20T17:49:54.442+0000 7f2f3ac8e700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.816989+0000 front 2023-07-20T17:49:30.817016+0000 (oldest deadline 2023-07-20T17:49:52.517188+0000) 2023-07-20T17:49:54.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:54 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:54.345+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:54.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:54 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:54.345+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:54.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:54 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:54.345+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:54.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:54 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:54.345+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:49:54.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:54 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:54.314+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:54.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:54 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:54.314+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:54.771 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:54 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:54.314+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:55.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:54 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:54.666+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:55.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:54 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:54.666+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:55.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:55 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:55.111+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:55.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:55 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:55.111+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:55.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:55 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:55.111+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:55.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:55 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:55.111+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:49:55.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:55 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:55.104+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:55.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:55 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:55.104+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:55.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:55 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:55.104+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:49:55.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:55 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:55.104+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:49:55.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:55 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:55.312+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:55.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:55 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:55.312+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:55.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:55 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:55.312+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:55.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:55 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:55.379+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:55.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:55 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:55.379+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:55.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:55 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:55.379+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:55.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:55 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:55.379+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:49:55.990 INFO:journalctl@ceph.osd.6.smithi049.stdout:Jul 20 17:49:55 smithi049 bash[134306]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-6 2023-07-20T17:49:56.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:55 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:55.665+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:56.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:55 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:55.665+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:56.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:56 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:56.139+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:56.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:56 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:56.139+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:56.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:56 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:56.139+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:56.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:56 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:56.139+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:49:56.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:56.080+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:56.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:56.080+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:56.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:56.080+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:49:56.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:56.080+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:49:56.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:56.404+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:56.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:56.404+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:56.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:56.404+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:56.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:56.404+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:49:56.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:56.272+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:56.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:56.272+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:56.771 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:56 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:56.272+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:56.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.6.service' 2023-07-20T17:49:56.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:49:56.938 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-20T17:49:56.938 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-20T17:49:56.939 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.7 2023-07-20T17:49:56.969 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:56.669+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:56.970 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:56.669+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:56.970 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:56 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:56.669+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:31.130076+0000 front 2023-07-20T17:49:31.130049+0000 (oldest deadline 2023-07-20T17:49:55.830064+0000) 2023-07-20T17:49:57.282 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:56 smithi079 systemd[1]: Stopping Ceph osd.7 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:49:57.282 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:57.066+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:57.282 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:57.066+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:57.283 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:57.066+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:49:57.283 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:57.066+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:49:57.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:57 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:57.168+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:57.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:57 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:57.168+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:57.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:57 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:57.168+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:57.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:57 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:57.168+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:49:57.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:57 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:57.416+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:57.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:57 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:57.416+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:57.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:57 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:57.416+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:57.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:57 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:57.416+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:49:57.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:57 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:57.259+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:57.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:57 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:57.259+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:57.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:57 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:57.259+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:57.568 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:57.279+0000 7f38be049700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:49:57.568 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:57.279+0000 7f38be049700 -1 osd.7 82 *** Got signal Terminated *** 2023-07-20T17:49:57.568 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:57.279+0000 7f38be049700 -1 osd.7 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:49:58.037 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:57.657+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:58.037 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:57.657+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:58.037 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:57 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:57.657+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:31.130076+0000 front 2023-07-20T17:49:31.130049+0000 (oldest deadline 2023-07-20T17:49:55.830064+0000) 2023-07-20T17:49:58.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:58.035+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:58.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:58.035+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:58.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:58.035+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:49:58.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:58.035+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:49:58.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:58 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:58.209+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:58.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:58 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:58.209+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:58.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:58 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:58.209+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:58.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:58 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:58.209+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:49:58.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:58 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:58.263+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:58.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:58 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:58.263+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:58.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:58 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:58.263+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:58.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:58 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:58.457+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:58.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:58 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:58.457+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:58.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:58 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:58.457+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:58.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:58 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:58.457+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:49:59.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:58.698+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:49:59.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:58.698+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:49:59.068 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:58.698+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:31.130076+0000 front 2023-07-20T17:49:31.130049+0000 (oldest deadline 2023-07-20T17:49:55.830064+0000) 2023-07-20T17:49:59.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:59 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:59.222+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:49:59.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:59 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:59.222+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:49:59.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:59 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:59.222+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:49:59.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:49:59 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:49:59.222+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:49:59.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:59.075+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:49:59.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:59.075+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:49:59.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:59.075+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:49:59.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:49:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:49:59.075+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:49:59.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:59 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:59.272+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:49:59.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:59 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:59.272+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:49:59.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:49:59 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:49:59.272+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:49:59.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:59 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:59.415+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:49:59.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:59 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:59.415+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:49:59.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:59 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:59.415+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:49:59.770 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:49:59 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:49:59.415+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:50:00.062 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:59.676+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:50:00.062 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:59.676+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:50:00.063 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:49:59 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:49:59.676+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:31.130076+0000 front 2023-07-20T17:49:31.130049+0000 (oldest deadline 2023-07-20T17:49:55.830064+0000) 2023-07-20T17:50:00.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:00.059+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:00.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:00.059+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:00.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:00.059+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:00.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:00.059+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:00.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:00 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:00.183+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:00.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:00 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:00.183+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:00.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:00 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:00.183+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:00.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:00 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:00.183+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:00.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:00 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:00.258+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:00.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:00 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:00.258+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:00.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:00 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:00.258+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:00.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:00 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:00.418+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:50:00.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:00 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:00.418+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:50:00.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:00 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:00.418+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:50:00.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:00 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:00.418+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:50:00.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:00 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:00.418+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.152200+0000 front 2023-07-20T17:49:39.152180+0000 (oldest deadline 2023-07-20T17:50:00.252284+0000) 2023-07-20T17:50:01.052 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:50:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:50:00.639+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:50:01.053 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:50:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:50:00.639+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:50:01.053 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:50:00 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:50:00.639+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:31.130076+0000 front 2023-07-20T17:49:31.130049+0000 (oldest deadline 2023-07-20T17:49:55.830064+0000) 2023-07-20T17:50:01.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:01 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:01.049+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:01.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:01 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:01.049+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:01.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:01 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:01.049+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:01.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:01 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:01.049+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:01.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:01 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:01.145+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:01.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:01 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:01.145+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:01.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:01 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:01.145+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:01.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:01 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:01.145+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:01.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:01 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:01.285+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:01.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:01 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:01.285+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:01.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:01 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:01.285+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:01.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:01 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:01.371+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:50:01.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:01 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:01.371+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:50:01.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:01 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:01.371+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:50:01.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:01 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:01.371+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:50:01.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:01 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:01.371+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.152200+0000 front 2023-07-20T17:49:39.152180+0000 (oldest deadline 2023-07-20T17:50:00.252284+0000) 2023-07-20T17:50:02.065 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:50:01 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:50:01.662+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.727018+0000 front 2023-07-20T17:49:16.727160+0000 (oldest deadline 2023-07-20T17:49:37.827218+0000) 2023-07-20T17:50:02.065 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:50:01 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:50:01.662+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.328809+0000 front 2023-07-20T17:49:22.328842+0000 (oldest deadline 2023-07-20T17:49:47.628956+0000) 2023-07-20T17:50:02.065 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:50:01 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7[114176]: 2023-07-20T17:50:01.662+0000 7f38b763a700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:31.130076+0000 front 2023-07-20T17:49:31.130049+0000 (oldest deadline 2023-07-20T17:49:55.830064+0000) 2023-07-20T17:50:02.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:02 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:02.062+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:02.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:02 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:02.062+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:02.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:02 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:02.062+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:02.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:02 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:02.062+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:02.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:02 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:02.099+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:02.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:02 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:02.099+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:02.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:02 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:02.099+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:02.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:02 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:02.099+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:02.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:02 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:02.099+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:02.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:02 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:02.323+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:02.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:02 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:02.323+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:02.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:02 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:02.323+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:02.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:02 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:02.325+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:50:02.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:02 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:02.325+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:50:02.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:02 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:02.325+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:50:02.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:02 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:02.325+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:50:02.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:02 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:02.325+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.152200+0000 front 2023-07-20T17:49:39.152180+0000 (oldest deadline 2023-07-20T17:50:00.252284+0000) 2023-07-20T17:50:03.065 INFO:journalctl@ceph.osd.7.smithi079.stdout:Jul 20 17:50:02 smithi079 bash[125663]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-7 2023-07-20T17:50:03.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:03 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:03.060+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:03.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:03 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:03.060+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:03.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:03 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:03.060+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:03.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:03 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:03.060+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:03.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:03 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:03.147+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:03.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:03 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:03.147+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:03.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:03 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:03.147+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:03.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:03 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:03.147+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:03.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:03 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:03.147+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:03.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:03 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:03.326+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:50:03.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:03 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:03.326+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:50:03.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:03 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:03.326+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:50:03.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:03 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:03.326+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:50:03.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:03 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:03.326+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.152200+0000 front 2023-07-20T17:49:39.152180+0000 (oldest deadline 2023-07-20T17:50:00.252284+0000) 2023-07-20T17:50:03.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:03 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:03.280+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:03.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:03 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:03.280+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:03.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:03 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:03.280+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:03.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:03 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:03.280+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:04.068 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:04 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:04.035+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:04.069 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:04 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:04.035+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:04.069 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:04 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:04.035+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:04.069 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:04 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:04.035+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:04.137 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.7.service' 2023-07-20T17:50:04.218 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:50:04.219 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-20T17:50:04.219 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-20T17:50:04.220 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.8 2023-07-20T17:50:04.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:04.137+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:04.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:04.137+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:04.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:04.137+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:04.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:04.137+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:04.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:04 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:04.137+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:04.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:04 smithi097 systemd[1]: Stopping Ceph osd.8 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:50:04.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:04.319+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:50:04.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:04.319+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:50:04.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:04.319+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:50:04.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:04.319+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:50:04.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:04.319+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.152200+0000 front 2023-07-20T17:49:39.152180+0000 (oldest deadline 2023-07-20T17:50:00.252284+0000) 2023-07-20T17:50:04.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:04.280+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:04.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:04.280+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:04.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:04.280+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:04.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:04.280+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:05.018 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:04.589+0000 7f260bd7c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:50:05.018 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:04.589+0000 7f260bd7c700 -1 osd.8 82 *** Got signal Terminated *** 2023-07-20T17:50:05.019 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:04 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:04.589+0000 7f260bd7c700 -1 osd.8 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:50:05.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:05 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:05.037+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:05.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:05 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:05.037+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:05.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:05 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:05.037+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:05.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:05 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:05.037+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:05.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:05 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:05.037+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:05.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:05 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:05.162+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:05.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:05 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:05.162+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:05.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:05 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:05.162+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:05.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:05 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:05.162+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:05.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:05 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:05.162+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:05.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:05 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:05.283+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:50:05.768 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:05 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:05.283+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:50:05.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:05 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:05.283+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:50:05.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:05 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:05.283+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:50:05.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:05 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:05.283+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.152200+0000 front 2023-07-20T17:49:39.152180+0000 (oldest deadline 2023-07-20T17:50:00.252284+0000) 2023-07-20T17:50:05.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:05 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:05.276+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:05.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:05 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:05.276+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:05.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:05 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:05.276+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:05.770 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:05 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:05.276+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:06.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:06 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:06.053+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:06.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:06 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:06.053+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:06.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:06 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:06.053+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:06.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:06 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:06.053+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:06.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:06 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:06.053+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:06.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:06 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:06.176+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:06.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:06 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:06.176+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:06.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:06 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:06.176+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:06.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:06 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:06.176+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:06.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:06 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:06.176+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:06.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:06 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:06.254+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:50:06.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:06 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:06.254+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:50:06.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:06 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:06.254+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:50:06.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:06 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:06.254+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:50:06.520 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:06 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:06.254+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.152200+0000 front 2023-07-20T17:49:39.152180+0000 (oldest deadline 2023-07-20T17:50:00.252284+0000) 2023-07-20T17:50:06.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:06 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:06.323+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:06.521 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:06 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:06.323+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:06.521 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:06 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:06.323+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:06.521 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:06 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:06.323+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:07.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:07 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:07.163+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:07.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:07 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:07.163+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:07.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:07 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:07.163+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:07.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:07 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:07.163+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:07.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:07 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:07.163+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:07.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:07 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:07.262+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:50:07.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:07 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:07.262+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:50:07.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:07 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:07.262+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:50:07.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:07 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:07.262+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:50:07.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:07 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:07.262+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.152200+0000 front 2023-07-20T17:49:39.152180+0000 (oldest deadline 2023-07-20T17:50:00.252284+0000) 2023-07-20T17:50:07.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:07 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:07.312+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:07.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:07 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:07.312+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:07.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:07 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:07.312+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:07.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:07 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:07.312+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:07.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:07 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:07.088+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:07.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:07 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:07.088+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:07.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:07 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:07.088+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:07.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:07 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:07.088+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:07.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:07 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:07.088+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:08.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:08 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:08.118+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:08.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:08 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:08.118+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:08.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:08 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:08.118+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:08.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:08 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:08.118+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:08.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:08 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:08.118+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:08.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:08 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:08.249+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:50:08.518 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:08 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:08.249+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:50:08.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:08 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:08.249+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:50:08.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:08 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:08.249+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:50:08.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:08 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:08.249+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.152200+0000 front 2023-07-20T17:49:39.152180+0000 (oldest deadline 2023-07-20T17:50:00.252284+0000) 2023-07-20T17:50:08.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:08 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:08.328+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:08.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:08 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:08.328+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:08.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:08 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:08.328+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:08.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:08 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:08.328+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:08.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:08 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:08.109+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:08.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:08 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:08.109+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:08.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:08 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:08.109+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:08.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:08 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:08.109+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:08.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:08 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:08.109+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:09.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:09 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:09.119+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:09.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:09 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:09.119+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:09.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:09 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:09.119+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:09.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:09 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:09.119+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:09.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:09 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:09.119+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:09.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:09 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:09.101+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:09.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:09 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:09.101+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:09.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:09 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:09.101+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:09.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:09 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:09.101+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:09.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:09 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:09.101+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:09.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:09 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:09.315+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:09.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:09 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:09.315+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:09.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:09 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:09.315+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:09.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:09 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:09.315+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:09.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:09 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:09.265+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:06.947172+0000 front 2023-07-20T17:49:06.947330+0000 (oldest deadline 2023-07-20T17:49:32.247381+0000) 2023-07-20T17:50:09.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:09 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:09.265+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:16.948786+0000 front 2023-07-20T17:49:16.948604+0000 (oldest deadline 2023-07-20T17:49:40.448682+0000) 2023-07-20T17:50:09.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:09 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:09.265+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.949758+0000 front 2023-07-20T17:49:23.949843+0000 (oldest deadline 2023-07-20T17:49:45.049880+0000) 2023-07-20T17:50:09.769 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:09 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:09.265+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:27.951076+0000 front 2023-07-20T17:49:27.950984+0000 (oldest deadline 2023-07-20T17:49:53.251071+0000) 2023-07-20T17:50:09.770 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:09 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8[115074]: 2023-07-20T17:50:09.265+0000 7f2607371700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.152200+0000 front 2023-07-20T17:49:39.152180+0000 (oldest deadline 2023-07-20T17:50:00.252284+0000) 2023-07-20T17:50:10.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:10 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:10.108+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:10.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:10 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:10.108+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:10.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:10 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:10.108+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:10.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:10 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:10.108+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:10.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:10 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:10.108+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:10.519 INFO:journalctl@ceph.osd.8.smithi097.stdout:Jul 20 17:50:10 smithi097 bash[126842]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-8 2023-07-20T17:50:10.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:10 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:10.336+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:10.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:10 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:10.336+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:10.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:10 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:10.336+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:10.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:10 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:10.336+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:10.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:10 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:10.108+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:10.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:10 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:10.108+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:10.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:10 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:10.108+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:10.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:10 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:10.108+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:10.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:10 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:10.108+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:10.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:10 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:10.108+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:11.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:11 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:11.077+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:11.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:11 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:11.077+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:11.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:11 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:11.077+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:11.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:11 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:11.077+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:11.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:11 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:11.077+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:11.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:11 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:11.083+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:11.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:11 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:11.083+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:11.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:11 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:11.083+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:11.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:11 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:11.083+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:11.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:11 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:11.083+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:11.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:11 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:11.083+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:11.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:11 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:11.303+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:11.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:11 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:11.303+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:11.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:11 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:11.303+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:11.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:11 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:11.303+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:11.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.8.service' 2023-07-20T17:50:11.830 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:50:11.830 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-20T17:50:11.831 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-20T17:50:11.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.9 2023-07-20T17:50:12.115 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:11 smithi049 systemd[1]: Stopping Ceph osd.9 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:50:12.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:12 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:12.042+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:12.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:12 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:12.042+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:12.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:12 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:12.042+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:12.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:12 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:12.042+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:12.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:12 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:12.042+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:12.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:12 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:12.042+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:12.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:12 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:12.112+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:12.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:12 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:12.112+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:12.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:12 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:12.112+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:12.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:12 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:12.112+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:12.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:12 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:12.112+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:12.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:12 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:12.112+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:43.028692+0000 front 2023-07-20T17:49:43.028583+0000 (oldest deadline 2023-07-20T17:50:11.828928+0000) 2023-07-20T17:50:12.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:12 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:12.196+0000 7eff22aaf700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:50:12.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:12 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:12.196+0000 7eff22aaf700 -1 osd.9 82 *** Got signal Terminated *** 2023-07-20T17:50:12.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:12 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:12.196+0000 7eff22aaf700 -1 osd.9 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:50:12.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:12 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:12.276+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:12.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:12 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:12.276+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:12.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:12 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:12.276+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:12.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:12 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:12.276+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:12.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:12 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:12.276+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:13.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:13 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:13.047+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:13.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:13 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:13.047+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:13.318 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:13 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:13.047+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:13.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:13 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:13.047+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:13.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:13 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:13.047+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:13.319 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:13 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:13.047+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:13.490 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:13 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:13.105+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:13.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:13 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:13.105+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:13.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:13 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:13.105+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:13.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:13 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:13.105+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:13.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:13 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:13.105+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:13.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:13 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:13.105+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:43.028692+0000 front 2023-07-20T17:49:43.028583+0000 (oldest deadline 2023-07-20T17:50:11.828928+0000) 2023-07-20T17:50:13.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:13 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:13.245+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:13.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:13 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:13.245+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:13.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:13 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:13.245+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:13.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:13 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:13.245+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:13.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:13 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:13.245+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:14.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:14 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:14.141+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:14.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:14 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:14.141+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:14.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:14 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:14.141+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:14.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:14 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:14.141+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:14.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:14 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:14.141+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:14.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:14 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:14.141+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:43.028692+0000 front 2023-07-20T17:49:43.028583+0000 (oldest deadline 2023-07-20T17:50:11.828928+0000) 2023-07-20T17:50:14.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:14 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:14.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:14.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:14 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:14.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:14.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:14 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:14.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:14.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:14 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:14.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:14.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:14 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:14.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:14.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:14 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:14.068+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:14.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:14 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:14.068+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:14.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:14 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:14.068+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:14.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:14 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:14.068+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:14.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:14 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:14.068+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:14.570 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:14 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:14.068+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:15.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:15 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:15.145+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:15.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:15 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:15.145+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:15.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:15 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:15.145+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:15.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:15 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:15.145+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:15.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:15 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:15.145+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:15.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:15 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:15.145+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:43.028692+0000 front 2023-07-20T17:49:43.028583+0000 (oldest deadline 2023-07-20T17:50:11.828928+0000) 2023-07-20T17:50:15.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:15 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:15.181+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:15.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:15 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:15.181+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:15.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:15 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:15.181+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:15.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:15 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:15.181+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:15.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:15 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:15.181+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:15.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:15 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:15.115+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:15.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:15 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:15.115+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:15.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:15 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:15.115+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:15.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:15 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:15.115+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:15.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:15 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:15.115+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:15.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:15 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:15.115+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:16.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:16 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:16.172+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:16.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:16 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:16.172+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:16.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:16 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:16.172+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:16.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:16 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:16.172+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:16.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:16 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:16.172+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:16.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:16 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:16.172+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:43.028692+0000 front 2023-07-20T17:49:43.028583+0000 (oldest deadline 2023-07-20T17:50:11.828928+0000) 2023-07-20T17:50:16.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:16 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:16.172+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:51.829405+0000 front 2023-07-20T17:49:51.829459+0000 (oldest deadline 2023-07-20T17:50:15.329528+0000) 2023-07-20T17:50:16.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:16 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:16.209+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:16.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:16 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:16.209+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:16.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:16 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:16.209+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:16.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:16 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:16.210+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:16.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:16 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:16.210+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:16.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:16 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:16.210+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:16.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:16 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:16.140+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:16.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:16 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:16.140+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:16.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:16 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:16.140+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:16.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:16 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:16.140+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:16.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:16 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:16.140+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:16.570 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:16 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:16.140+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:16.570 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:16 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:16.140+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:53.811121+0000 front 2023-07-20T17:49:53.811232+0000 (oldest deadline 2023-07-20T17:50:16.111001+0000) 2023-07-20T17:50:17.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:17 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:17.148+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.221050+0000 front 2023-07-20T17:49:08.221270+0000 (oldest deadline 2023-07-20T17:49:30.521164+0000) 2023-07-20T17:50:17.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:17 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:17.148+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.123388+0000 front 2023-07-20T17:49:15.123221+0000 (oldest deadline 2023-07-20T17:49:39.223264+0000) 2023-07-20T17:50:17.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:17 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:17.148+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.324468+0000 front 2023-07-20T17:49:23.324447+0000 (oldest deadline 2023-07-20T17:49:45.024564+0000) 2023-07-20T17:50:17.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:17 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:17.148+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:29.626267+0000 front 2023-07-20T17:49:29.626299+0000 (oldest deadline 2023-07-20T17:49:54.926393+0000) 2023-07-20T17:50:17.491 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:17 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:17.148+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.027577+0000 front 2023-07-20T17:49:39.027691+0000 (oldest deadline 2023-07-20T17:50:01.927630+0000) 2023-07-20T17:50:17.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:17 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:17.148+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:43.028692+0000 front 2023-07-20T17:49:43.028583+0000 (oldest deadline 2023-07-20T17:50:11.828928+0000) 2023-07-20T17:50:17.492 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:17 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9[126031]: 2023-07-20T17:50:17.148+0000 7eff1e8a5700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:51.829405+0000 front 2023-07-20T17:49:51.829459+0000 (oldest deadline 2023-07-20T17:50:15.329528+0000) 2023-07-20T17:50:17.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:17 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:17.237+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:17.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:17 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:17.237+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:17.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:17 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:17.237+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:17.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:17 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:17.237+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:17.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:17 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:17.237+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:17.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:17 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:17.237+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:17.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:17 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:17.189+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:17.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:17 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:17.189+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:17.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:17 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:17.189+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:17.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:17 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:17.189+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:17.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:17 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:17.189+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:17.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:17 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:17.189+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:17.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:17 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:17.189+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:53.811121+0000 front 2023-07-20T17:49:53.811232+0000 (oldest deadline 2023-07-20T17:50:16.111001+0000) 2023-07-20T17:50:18.078 INFO:journalctl@ceph.osd.9.smithi049.stdout:Jul 20 17:50:17 smithi049 bash[134584]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-9 2023-07-20T17:50:18.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:18 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:18.214+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:18.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:18 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:18.214+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:18.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:18 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:18.214+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:18.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:18 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:18.214+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:18.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:18 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:18.214+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:18.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:18 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:18.214+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:18.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:18 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:18.220+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:18.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:18 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:18.220+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:18.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:18 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:18.220+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:18.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:18 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:18.220+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:18.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:18 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:18.220+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:18.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:18 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:18.220+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:18.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:18 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:18.220+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:53.811121+0000 front 2023-07-20T17:49:53.811232+0000 (oldest deadline 2023-07-20T17:50:16.111001+0000) 2023-07-20T17:50:19.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.9.service' 2023-07-20T17:50:19.497 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:50:19.498 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-20T17:50:19.498 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-20T17:50:19.499 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.10 2023-07-20T17:50:19.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:19 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:19.189+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:19.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:19 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:19.189+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:19.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:19 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:19.189+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:19.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:19 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:19.189+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:19.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:19 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:19.189+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:19.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:19 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:19.189+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:19.528 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:19 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:19.180+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:19.528 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:19 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:19.180+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:19.528 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:19 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:19.180+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:19.528 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:19 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:19.180+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:19.529 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:19 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:19.180+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:19.529 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:19 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:19.180+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:19.529 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:19 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:19.180+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:53.811121+0000 front 2023-07-20T17:49:53.811232+0000 (oldest deadline 2023-07-20T17:50:16.111001+0000) 2023-07-20T17:50:19.818 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:19 smithi079 systemd[1]: Stopping Ceph osd.10 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:50:20.161 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:19 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:19.840+0000 7f04feda2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:50:20.162 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:19 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:19.840+0000 7f04feda2700 -1 osd.10 82 *** Got signal Terminated *** 2023-07-20T17:50:20.162 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:19 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:19.840+0000 7f04feda2700 -1 osd.10 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:50:20.162 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:20 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:20.159+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:20.162 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:20 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:20.159+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:20.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:20 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:20.170+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:20.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:20 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:20.170+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:20.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:20 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:20.170+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:20.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:20 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:20.170+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:20.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:20 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:20.170+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:20.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:20 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:20.170+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:20.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:20 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:20.159+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:20.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:20 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:20.159+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:20.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:20 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:20.159+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:20.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:20 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:20.159+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:20.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:20 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:20.159+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:53.811121+0000 front 2023-07-20T17:49:53.811232+0000 (oldest deadline 2023-07-20T17:50:16.111001+0000) 2023-07-20T17:50:21.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:21 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:21.208+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:21.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:21 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:21.208+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:21.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:21 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:21.208+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:21.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:21 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:21.208+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:21.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:21 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:21.208+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:21.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:21 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:21.208+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:21.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:21 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:21.131+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:21.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:21 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:21.131+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:21.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:21 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:21.131+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:21.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:21 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:21.131+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:21.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:21 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:21.131+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:21.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:21 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:21.131+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:21.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:21 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:21.131+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:53.811121+0000 front 2023-07-20T17:49:53.811232+0000 (oldest deadline 2023-07-20T17:50:16.111001+0000) 2023-07-20T17:50:22.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:22 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:22.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:22.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:22 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:22.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:22.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:22 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:22.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:22.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:22 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:22.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:22.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:22 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:22.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:22.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:22 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:22.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:22.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:22 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:22.130+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:22.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:22 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:22.130+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:22.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:22 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:22.130+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:22.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:22 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:22.130+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:22.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:22 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:22.130+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:22.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:22 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:22.130+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:22.570 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:22 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:22.130+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:53.811121+0000 front 2023-07-20T17:49:53.811232+0000 (oldest deadline 2023-07-20T17:50:16.111001+0000) 2023-07-20T17:50:23.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:23 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:23.229+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:23.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:23 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:23.229+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:23.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:23 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:23.229+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:23.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:23 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:23.229+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:23.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:23 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:23.229+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:23.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:23 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:23.229+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:23.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:23 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:23.110+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:23.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:23 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:23.110+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:23.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:23 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:23.110+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:23.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:23 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:23.110+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:23.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:23 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:23.110+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:23.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:23 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:23.110+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:23.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:23 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:23.110+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:53.811121+0000 front 2023-07-20T17:49:53.811232+0000 (oldest deadline 2023-07-20T17:50:16.111001+0000) 2023-07-20T17:50:24.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:24 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:24.193+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:24.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:24 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:24.193+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:24.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:24 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:24.193+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:24.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:24 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:24.193+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:24.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:24 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:24.193+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:24.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:24 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:24.193+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:24.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:24 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:24.193+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2023-07-20T17:50:01.830654+0000 front 2023-07-20T17:50:01.830670+0000 (oldest deadline 2023-07-20T17:50:23.530601+0000) 2023-07-20T17:50:24.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:24 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:24.151+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:07.303199+0000 front 2023-07-20T17:49:07.303112+0000 (oldest deadline 2023-07-20T17:49:31.403251+0000) 2023-07-20T17:50:24.568 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:24 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:24.151+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:15.404582+0000 front 2023-07-20T17:49:15.404814+0000 (oldest deadline 2023-07-20T17:49:41.304788+0000) 2023-07-20T17:50:24.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:24 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:24.151+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:23.606072+0000 front 2023-07-20T17:49:23.606204+0000 (oldest deadline 2023-07-20T17:49:49.505995+0000) 2023-07-20T17:50:24.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:24 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:24.151+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2023-07-20T17:49:30.607128+0000 front 2023-07-20T17:49:30.607197+0000 (oldest deadline 2023-07-20T17:49:54.607477+0000) 2023-07-20T17:50:24.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:24 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:24.151+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:39.109302+0000 front 2023-07-20T17:49:39.109348+0000 (oldest deadline 2023-07-20T17:50:04.409450+0000) 2023-07-20T17:50:24.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:24 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:24.151+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:44.409955+0000 front 2023-07-20T17:49:44.409996+0000 (oldest deadline 2023-07-20T17:50:09.710051+0000) 2023-07-20T17:50:24.569 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:24 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10[119652]: 2023-07-20T17:50:24.151+0000 7f04f7b92700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:53.811121+0000 front 2023-07-20T17:49:53.811232+0000 (oldest deadline 2023-07-20T17:50:16.111001+0000) 2023-07-20T17:50:25.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:25 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:25.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:25.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:25 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:25.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:25.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:25 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:25.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:25.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:25 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:25.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:25.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:25 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:25.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:25.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:25 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:25.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:25.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:25 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:25.198+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2023-07-20T17:50:01.830654+0000 front 2023-07-20T17:50:01.830670+0000 (oldest deadline 2023-07-20T17:50:23.530601+0000) 2023-07-20T17:50:25.731 INFO:journalctl@ceph.osd.10.smithi079.stdout:Jul 20 17:50:25 smithi079 bash[125927]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-10 2023-07-20T17:50:26.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:26 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:26.150+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:26.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:26 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:26.150+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:26.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:26 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:26.150+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:26.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:26 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:26.150+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:26.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:26 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:26.150+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:26.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:26 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:26.150+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:26.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:26 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:26.150+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2023-07-20T17:50:01.830654+0000 front 2023-07-20T17:50:01.830670+0000 (oldest deadline 2023-07-20T17:50:23.530601+0000) 2023-07-20T17:50:26.892 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.10.service' 2023-07-20T17:50:26.972 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:50:26.974 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-20T17:50:26.976 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-20T17:50:26.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.11 2023-07-20T17:50:27.269 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:27 smithi097 systemd[1]: Stopping Ceph osd.11 for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:50:27.269 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:27 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:27.159+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:27.269 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:27 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:27.159+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:27.269 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:27 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:27.159+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:27.269 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:27 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:27.159+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:27.270 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:27 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:27.159+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:27.270 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:27 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:27.159+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:27.270 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:27 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:27.159+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2023-07-20T17:50:01.830654+0000 front 2023-07-20T17:50:01.830670+0000 (oldest deadline 2023-07-20T17:50:23.530601+0000) 2023-07-20T17:50:27.768 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:27 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:27.328+0000 7fc5b6319700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:50:27.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:27 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:27.328+0000 7fc5b6319700 -1 osd.11 82 *** Got signal Terminated *** 2023-07-20T17:50:27.769 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:27 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:27.328+0000 7fc5b6319700 -1 osd.11 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T17:50:28.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:28 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:28.173+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:28.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:28 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:28.173+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:28.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:28 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:28.173+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:28.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:28 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:28.173+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:28.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:28 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:28.173+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:28.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:28 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:28.173+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:28.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:28 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:28.173+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2023-07-20T17:50:01.830654+0000 front 2023-07-20T17:50:01.830670+0000 (oldest deadline 2023-07-20T17:50:23.530601+0000) 2023-07-20T17:50:29.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:29 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:29.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:29.521 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:29 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:29.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:29.521 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:29 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:29.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:29.521 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:29 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:29.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:29.521 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:29 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:29.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:29.521 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:29 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:29.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:29.521 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:29 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:29.215+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2023-07-20T17:50:01.830654+0000 front 2023-07-20T17:50:01.830670+0000 (oldest deadline 2023-07-20T17:50:23.530601+0000) 2023-07-20T17:50:30.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:30 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:30.224+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:30.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:30 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:30.224+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:30.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:30 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:30.224+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:30.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:30 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:30.224+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:30.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:30 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:30.224+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:30.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:30 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:30.224+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:30.520 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:30 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:30.224+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2023-07-20T17:50:01.830654+0000 front 2023-07-20T17:50:01.830670+0000 (oldest deadline 2023-07-20T17:50:23.530601+0000) 2023-07-20T17:50:31.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:31 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:31.178+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:31.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:31 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:31.178+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:31.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:31 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:31.179+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:31.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:31 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:31.179+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:31.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:31 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:31.179+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:31.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:31 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:31.179+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:31.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:31 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:31.179+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2023-07-20T17:50:01.830654+0000 front 2023-07-20T17:50:01.830670+0000 (oldest deadline 2023-07-20T17:50:23.530601+0000) 2023-07-20T17:50:32.518 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:32 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:32.199+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2023-07-20T17:49:08.821691+0000 front 2023-07-20T17:49:08.821732+0000 (oldest deadline 2023-07-20T17:49:32.321698+0000) 2023-07-20T17:50:32.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:32 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:32.199+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6804 osd.1 since back 2023-07-20T17:49:12.322465+0000 front 2023-07-20T17:49:12.322273+0000 (oldest deadline 2023-07-20T17:49:37.622332+0000) 2023-07-20T17:50:32.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:32 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:32.199+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6804 osd.2 since back 2023-07-20T17:49:22.923420+0000 front 2023-07-20T17:49:22.923356+0000 (oldest deadline 2023-07-20T17:49:45.223476+0000) 2023-07-20T17:50:32.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:32 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:32.199+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6812 osd.4 since back 2023-07-20T17:49:37.926774+0000 front 2023-07-20T17:49:37.926786+0000 (oldest deadline 2023-07-20T17:50:02.626900+0000) 2023-07-20T17:50:32.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:32 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:32.199+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-07-20T17:49:46.727937+0000 front 2023-07-20T17:49:46.727900+0000 (oldest deadline 2023-07-20T17:50:12.027978+0000) 2023-07-20T17:50:32.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:32 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:32.199+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2023-07-20T17:49:52.529197+0000 front 2023-07-20T17:49:52.529213+0000 (oldest deadline 2023-07-20T17:50:16.029052+0000) 2023-07-20T17:50:32.519 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:32 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11[120616]: 2023-07-20T17:50:32.199+0000 7fc5ae908700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.79:6820 osd.7 since back 2023-07-20T17:50:01.830654+0000 front 2023-07-20T17:50:01.830670+0000 (oldest deadline 2023-07-20T17:50:23.530601+0000) 2023-07-20T17:50:33.268 INFO:journalctl@ceph.osd.11.smithi097.stdout:Jul 20 17:50:33 smithi097 bash[127119]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-osd-11 2023-07-20T17:50:34.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@osd.11.service' 2023-07-20T17:50:34.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:50:34.516 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-20T17:50:34.517 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-20T17:50:34.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.a 2023-07-20T17:50:34.887 INFO:journalctl@ceph.mds.a.smithi049.stdout:Jul 20 17:50:34 smithi049 systemd[1]: Stopping Ceph mds.a for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:50:35.240 INFO:journalctl@ceph.mds.a.smithi049.stdout:Jul 20 17:50:34 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-a[131380]: 2023-07-20T17:50:34.883+0000 7f92cafa7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:50:35.241 INFO:journalctl@ceph.mds.a.smithi049.stdout:Jul 20 17:50:34 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-a[131380]: 2023-07-20T17:50:34.883+0000 7f92cafa7700 -1 mds.a *** got signal Terminated *** 2023-07-20T17:50:37.991 INFO:journalctl@ceph.mds.a.smithi049.stdout:Jul 20 17:50:37 smithi049 bash[134849]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-a 2023-07-20T17:50:38.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.a.service' 2023-07-20T17:50:38.186 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:50:38.187 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-20T17:50:38.187 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-20T17:50:38.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.d 2023-07-20T17:50:38.491 INFO:journalctl@ceph.mds.d.smithi049.stdout:Jul 20 17:50:38 smithi049 systemd[1]: Stopping Ceph mds.d for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:50:38.990 INFO:journalctl@ceph.mds.d.smithi049.stdout:Jul 20 17:50:38 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-d[130385]: 2023-07-20T17:50:38.565+0000 7ff7a82b8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:50:38.991 INFO:journalctl@ceph.mds.d.smithi049.stdout:Jul 20 17:50:38 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-d[130385]: 2023-07-20T17:50:38.565+0000 7ff7a82b8700 -1 mds.d *** got signal Terminated *** 2023-07-20T17:50:43.156 INFO:journalctl@ceph.mds.d.smithi049.stdout:Jul 20 17:50:42 smithi049 bash[134985]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-d 2023-07-20T17:50:43.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.d.service' 2023-07-20T17:50:43.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:50:43.197 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-20T17:50:43.198 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-20T17:50:43.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.g 2023-07-20T17:50:43.491 INFO:journalctl@ceph.mds.g.smithi049.stdout:Jul 20 17:50:43 smithi049 systemd[1]: Stopping Ceph mds.g for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:50:43.991 INFO:journalctl@ceph.mds.g.smithi049.stdout:Jul 20 17:50:43 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-g[130997]: 2023-07-20T17:50:43.573+0000 7ff5bb0c7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:50:43.991 INFO:journalctl@ceph.mds.g.smithi049.stdout:Jul 20 17:50:43 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-g[130997]: 2023-07-20T17:50:43.573+0000 7ff5bb0c7700 -1 mds.g *** got signal Terminated *** 2023-07-20T17:50:45.439 INFO:journalctl@ceph.mds.g.smithi049.stdout:Jul 20 17:50:45 smithi049 bash[135131]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-g 2023-07-20T17:50:45.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.g.service' 2023-07-20T17:50:45.479 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:50:45.479 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-20T17:50:45.480 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-20T17:50:45.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.j 2023-07-20T17:50:45.740 INFO:journalctl@ceph.mds.j.smithi049.stdout:Jul 20 17:50:45 smithi049 systemd[1]: Stopping Ceph mds.j for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:50:46.240 INFO:journalctl@ceph.mds.j.smithi049.stdout:Jul 20 17:50:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-j[129999]: 2023-07-20T17:50:45.858+0000 7f079589d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:50:46.241 INFO:journalctl@ceph.mds.j.smithi049.stdout:Jul 20 17:50:45 smithi049 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-j[129999]: 2023-07-20T17:50:45.858+0000 7f079589d700 -1 mds.j *** got signal Terminated *** 2023-07-20T17:50:49.228 INFO:journalctl@ceph.mds.j.smithi049.stdout:Jul 20 17:50:49 smithi049 bash[135278]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-j 2023-07-20T17:50:49.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.j.service' 2023-07-20T17:50:49.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:50:49.480 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-20T17:50:49.481 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-20T17:50:49.481 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.b 2023-07-20T17:50:49.818 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:50:49 smithi079 systemd[1]: Stopping Ceph mds.b for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:50:50.318 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:50:49 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-b[123332]: 2023-07-20T17:50:49.824+0000 7f831f074700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:50:50.318 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:50:49 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-b[123332]: 2023-07-20T17:50:49.825+0000 7f831f074700 -1 mds.b *** got signal Terminated *** 2023-07-20T17:50:57.805 INFO:journalctl@ceph.mds.b.smithi079.stdout:Jul 20 17:50:57 smithi079 bash[126190]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-b 2023-07-20T17:50:57.812 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.b.service' 2023-07-20T17:50:57.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:50:57.851 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-20T17:50:57.852 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-20T17:50:57.852 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.e 2023-07-20T17:50:58.068 INFO:journalctl@ceph.mds.e.smithi079.stdout:Jul 20 17:50:57 smithi079 systemd[1]: Stopping Ceph mds.e for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:50:58.568 INFO:journalctl@ceph.mds.e.smithi079.stdout:Jul 20 17:50:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-e[122653]: 2023-07-20T17:50:58.223+0000 7f2ffea30700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:50:58.568 INFO:journalctl@ceph.mds.e.smithi079.stdout:Jul 20 17:50:58 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-e[122653]: 2023-07-20T17:50:58.223+0000 7f2ffea30700 -1 mds.e *** got signal Terminated *** 2023-07-20T17:51:01.688 INFO:journalctl@ceph.mds.e.smithi079.stdout:Jul 20 17:51:01 smithi079 bash[126338]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-e 2023-07-20T17:51:01.693 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.e.service' 2023-07-20T17:51:01.729 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:51:01.729 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-20T17:51:01.730 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-20T17:51:01.730 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.h 2023-07-20T17:51:02.069 INFO:journalctl@ceph.mds.h.smithi079.stdout:Jul 20 17:51:01 smithi079 systemd[1]: Stopping Ceph mds.h for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:51:02.568 INFO:journalctl@ceph.mds.h.smithi079.stdout:Jul 20 17:51:02 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-h[122992]: 2023-07-20T17:51:02.105+0000 7fb258ef2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:51:02.569 INFO:journalctl@ceph.mds.h.smithi079.stdout:Jul 20 17:51:02 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-h[122992]: 2023-07-20T17:51:02.105+0000 7fb258ef2700 -1 mds.h *** got signal Terminated *** 2023-07-20T17:51:05.172 INFO:journalctl@ceph.mds.h.smithi079.stdout:Jul 20 17:51:04 smithi079 bash[126489]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-h 2023-07-20T17:51:05.177 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.h.service' 2023-07-20T17:51:05.212 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:51:05.212 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-20T17:51:05.213 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-20T17:51:05.213 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.k 2023-07-20T17:51:05.568 INFO:journalctl@ceph.mds.k.smithi079.stdout:Jul 20 17:51:05 smithi079 systemd[1]: Stopping Ceph mds.k for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:51:06.068 INFO:journalctl@ceph.mds.k.smithi079.stdout:Jul 20 17:51:05 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-k[122314]: 2023-07-20T17:51:05.588+0000 7fdcfaaeb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:51:06.069 INFO:journalctl@ceph.mds.k.smithi079.stdout:Jul 20 17:51:05 smithi079 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-k[122314]: 2023-07-20T17:51:05.588+0000 7fdcfaaeb700 -1 mds.k *** got signal Terminated *** 2023-07-20T17:51:07.146 INFO:journalctl@ceph.mds.k.smithi079.stdout:Jul 20 17:51:07 smithi079 bash[126634]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-k 2023-07-20T17:51:07.351 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.k.service' 2023-07-20T17:51:07.383 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:51:07.384 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-20T17:51:07.384 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-20T17:51:07.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.c 2023-07-20T17:51:07.759 INFO:journalctl@ceph.mds.c.smithi097.stdout:Jul 20 17:51:07 smithi097 systemd[1]: Stopping Ceph mds.c for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:51:08.018 INFO:journalctl@ceph.mds.c.smithi097.stdout:Jul 20 17:51:07 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-c[124204]: 2023-07-20T17:51:07.755+0000 7f69452cb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:51:08.019 INFO:journalctl@ceph.mds.c.smithi097.stdout:Jul 20 17:51:07 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-c[124204]: 2023-07-20T17:51:07.755+0000 7f69452cb700 -1 mds.c *** got signal Terminated *** 2023-07-20T17:51:14.018 INFO:journalctl@ceph.mds.c.smithi097.stdout:Jul 20 17:51:13 smithi097 bash[127397]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-c 2023-07-20T17:51:14.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.c.service' 2023-07-20T17:51:14.213 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:51:14.214 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-20T17:51:14.214 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-20T17:51:14.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.f 2023-07-20T17:51:14.518 INFO:journalctl@ceph.mds.f.smithi097.stdout:Jul 20 17:51:14 smithi097 systemd[1]: Stopping Ceph mds.f for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:51:15.018 INFO:journalctl@ceph.mds.f.smithi097.stdout:Jul 20 17:51:14 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-f[123866]: 2023-07-20T17:51:14.612+0000 7f17475ec700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:51:15.019 INFO:journalctl@ceph.mds.f.smithi097.stdout:Jul 20 17:51:14 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-f[123866]: 2023-07-20T17:51:14.612+0000 7f17475ec700 -1 mds.f *** got signal Terminated *** 2023-07-20T17:51:19.449 INFO:journalctl@ceph.mds.f.smithi097.stdout:Jul 20 17:51:19 smithi097 bash[127550]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-f 2023-07-20T17:51:19.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.f.service' 2023-07-20T17:51:19.489 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:51:19.490 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-20T17:51:19.490 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-20T17:51:19.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.i 2023-07-20T17:51:19.768 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:51:19 smithi097 systemd[1]: Stopping Ceph mds.i for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:51:20.268 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:51:19 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-i[124544]: 2023-07-20T17:51:19.854+0000 7f7db3363700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:51:20.269 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:51:19 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-i[124544]: 2023-07-20T17:51:19.854+0000 7f7db3363700 -1 mds.i *** got signal Terminated *** 2023-07-20T17:51:24.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.i.service' 2023-07-20T17:51:24.153 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:51:23 smithi097 bash[127688]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-i 2023-07-20T17:51:24.153 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:51:24 smithi097 systemd[1]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.i.service: Succeeded. 2023-07-20T17:51:24.153 INFO:journalctl@ceph.mds.i.smithi097.stdout:Jul 20 17:51:24 smithi097 systemd[1]: Stopped Ceph mds.i for 8c050cea-2724-11ee-9b34-001a4aab830c. 2023-07-20T17:51:24.164 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:51:24.165 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-20T17:51:24.165 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-20T17:51:24.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.l 2023-07-20T17:51:24.519 INFO:journalctl@ceph.mds.l.smithi097.stdout:Jul 20 17:51:24 smithi097 systemd[1]: Stopping Ceph mds.l for 8c050cea-2724-11ee-9b34-001a4aab830c... 2023-07-20T17:51:25.019 INFO:journalctl@ceph.mds.l.smithi097.stdout:Jul 20 17:51:24 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-l[123525]: 2023-07-20T17:51:24.544+0000 7fb579e76700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T17:51:25.019 INFO:journalctl@ceph.mds.l.smithi097.stdout:Jul 20 17:51:24 smithi097 ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-l[123525]: 2023-07-20T17:51:24.544+0000 7fb579e76700 -1 mds.l *** got signal Terminated *** 2023-07-20T17:51:28.920 INFO:journalctl@ceph.mds.l.smithi097.stdout:Jul 20 17:51:28 smithi097 bash[127826]: ceph-8c050cea-2724-11ee-9b34-001a4aab830c-mds-l 2023-07-20T17:51:29.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c050cea-2724-11ee-9b34-001a4aab830c@mds.l.service' 2023-07-20T17:51:29.139 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T17:51:29.139 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-20T17:51:29.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8c050cea-2724-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T17:51:29.474 INFO:teuthology.orchestra.run.smithi049.stdout:Deleting cluster with fsid: 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:51:31.118 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8c050cea-2724-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T17:51:31.433 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:51:32.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8c050cea-2724-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T17:51:33.272 INFO:teuthology.orchestra.run.smithi097.stdout:Deleting cluster with fsid: 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:51:34.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T17:51:34.856 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T17:51:34.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T17:51:34.924 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-20T17:51:34.926 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/remote/smithi049/crash 2023-07-20T17:51:34.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T17:51:34.972 INFO:teuthology.orchestra.run.smithi049.stderr:tar: /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T17:51:34.972 INFO:teuthology.orchestra.run.smithi049.stderr:tar: Error is not recoverable: exiting now 2023-07-20T17:51:34.975 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/remote/smithi079/crash 2023-07-20T17:51:34.977 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T17:51:35.052 INFO:teuthology.orchestra.run.smithi079.stderr:tar: /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T17:51:35.052 INFO:teuthology.orchestra.run.smithi079.stderr:tar: Error is not recoverable: exiting now 2023-07-20T17:51:35.053 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/remote/smithi097/crash 2023-07-20T17:51:35.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T17:51:35.088 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T17:51:35.089 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-07-20T17:51:35.089 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-20T17:51:35.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-07-20T17:51:35.126 INFO:teuthology.orchestra.run.smithi049.stderr:grep: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph.log: No such file or directory 2023-07-20T17:51:35.128 INFO:tasks.cephadm:Compressing logs... 2023-07-20T17:51:35.128 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T17:51:35.170 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T17:51:35.173 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T17:51:35.198 INFO:teuthology.orchestra.run.smithi049.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-20T17:51:35.199 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T17:51:35.199 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.0.log 2023-07-20T17:51:35.201 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.g.log 2023-07-20T17:51:35.202 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.a.log 2023-07-20T17:51:35.202 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.6.log 2023-07-20T17:51:35.202 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.3.log 2023-07-20T17:51:35.202 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.6.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.3.log: 94.7% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.g.log.gz 2023-07-20T17:51:35.203 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/rbd-target-api’gzip : No such file or directory-5 2023-07-20T17:51:35.203 INFO:teuthology.orchestra.run.smithi097.stderr:--verbose -- /var/log/ceph/cephadm.log 2023-07-20T17:51:35.203 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.5.log 2023-07-20T17:51:35.203 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.2.log 2023-07-20T17:51:35.203 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.11.log 2023-07-20T17:51:35.204 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.5.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mgr.z.log 2023-07-20T17:51:35.204 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.8.log 2023-07-20T17:51:35.204 INFO:teuthology.orchestra.run.smithi097.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T17:51:35.204 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.f.log 2023-07-20T17:51:35.204 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.c.log 2023-07-20T17:51:35.205 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.8.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.f.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.c.log: 91.2% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mgr.z.log.gz 2023-07-20T17:51:35.205 INFO:teuthology.orchestra.run.smithi049.stderr: 93.6% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.a.log.gz 2023-07-20T17:51:35.206 INFO:teuthology.orchestra.run.smithi079.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-20T17:51:35.206 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T17:51:35.206 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.10.log 2023-07-20T17:51:35.206 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.h.log 2023-07-20T17:51:35.206 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.e.log 2023-07-20T17:51:35.207 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.10.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.4.log 2023-07-20T17:51:35.207 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.e.log: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T17:51:35.207 INFO:teuthology.orchestra.run.smithi097.stderr: 93.9% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.f.log.gz 2023-07-20T17:51:35.210 INFO:teuthology.orchestra.run.smithi049.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T17:51:35.210 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.d.log 2023-07-20T17:51:35.210 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mgr.x.log 2023-07-20T17:51:35.211 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.9.log 2023-07-20T17:51:35.211 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T17:51:35.211 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.9.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-volume.log: 94.7% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.d.log.gz 2023-07-20T17:51:35.211 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.1.log 2023-07-20T17:51:35.212 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.k.log 2023-07-20T17:51:35.212 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.1.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T17:51:35.212 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-volume.log: 94.0% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.e.log.gz 2023-07-20T17:51:35.212 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.7.log 2023-07-20T17:51:35.212 INFO:teuthology.orchestra.run.smithi079.stderr: 94.0% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.k.log.gz 2023-07-20T17:51:35.212 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mon.b.log 2023-07-20T17:51:35.212 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.7.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mon.b.log: 93.2% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.h.log.gz 2023-07-20T17:51:35.212 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.b.log 2023-07-20T17:51:35.213 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mgr.y.log 2023-07-20T17:51:35.213 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.i.log 2023-07-20T17:51:35.213 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mon.c.log 2023-07-20T17:51:35.214 INFO:teuthology.orchestra.run.smithi097.stderr: 93.5% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.c.log.gz 2023-07-20T17:51:35.214 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.i.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T17:51:35.214 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.l.log 2023-07-20T17:51:35.214 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.b.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mgr.y.log: 91.4% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T17:51:35.215 INFO:teuthology.orchestra.run.smithi079.stderr: 91.1% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mgr.y.log.gz 2023-07-20T17:51:35.215 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.l.log: 93.4% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.i.log.gz 2023-07-20T17:51:35.216 INFO:teuthology.orchestra.run.smithi049.stderr: 91.6% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T17:51:35.216 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.j.log 2023-07-20T17:51:35.217 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mon.a.log 2023-07-20T17:51:35.217 INFO:teuthology.orchestra.run.smithi097.stderr: 91.5% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T17:51:35.218 INFO:teuthology.orchestra.run.smithi097.stderr: 94.3% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.l.log.gz 2023-07-20T17:51:35.218 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.j.log: /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mon.a.log: 95.0% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.j.log.gz 2023-07-20T17:51:35.238 INFO:teuthology.orchestra.run.smithi079.stderr: 93.2% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mds.b.log.gz 2023-07-20T17:51:35.297 INFO:teuthology.orchestra.run.smithi097.stderr: 92.8% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.11.log.gz 2023-07-20T17:51:35.306 INFO:teuthology.orchestra.run.smithi097.stderr: 92.0% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mon.c.log.gz 2023-07-20T17:51:35.311 INFO:teuthology.orchestra.run.smithi079.stderr: 91.9% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mon.b.log.gz 2023-07-20T17:51:35.323 INFO:teuthology.orchestra.run.smithi049.stderr: 88.7% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mgr.x.log.gz 2023-07-20T17:51:35.341 INFO:teuthology.orchestra.run.smithi079.stderr: 92.0% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.4.log.gz 2023-07-20T17:51:35.344 INFO:teuthology.orchestra.run.smithi079.stderr: 92.2% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.10.log.gz 2023-07-20T17:51:35.355 INFO:teuthology.orchestra.run.smithi049.stderr: 92.0% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.6.log.gz 2023-07-20T17:51:35.365 INFO:teuthology.orchestra.run.smithi049.stderr: 92.2% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.9.log.gz 2023-07-20T17:51:35.369 INFO:teuthology.orchestra.run.smithi097.stderr: 91.8% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.5.log.gz 2023-07-20T17:51:35.375 INFO:teuthology.orchestra.run.smithi097.stderr: 92.1% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.8.log.gz 2023-07-20T17:51:35.401 INFO:teuthology.orchestra.run.smithi049.stderr: 91.8% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.3.log.gz 2023-07-20T17:51:35.404 INFO:teuthology.orchestra.run.smithi079.stderr: 92.0% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.7.log.gz 2023-07-20T17:51:35.406 INFO:teuthology.orchestra.run.smithi049.stderr: 92.0% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.0.log.gz 2023-07-20T17:51:35.419 INFO:teuthology.orchestra.run.smithi097.stderr: 91.8% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.2.log.gz 2023-07-20T17:51:35.421 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-07-20T17:51:35.421 INFO:teuthology.orchestra.run.smithi097.stderr:real 0m0.232s 2023-07-20T17:51:35.421 INFO:teuthology.orchestra.run.smithi097.stderr:user 0m0.769s 2023-07-20T17:51:35.421 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m0.048s 2023-07-20T17:51:35.460 INFO:teuthology.orchestra.run.smithi079.stderr: 91.7% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-osd.1.log.gz 2023-07-20T17:51:35.462 INFO:teuthology.orchestra.run.smithi079.stderr: 2023-07-20T17:51:35.462 INFO:teuthology.orchestra.run.smithi079.stderr:real 0m0.273s 2023-07-20T17:51:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:user 0m0.870s 2023-07-20T17:51:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m0.050s 2023-07-20T17:51:35.566 INFO:teuthology.orchestra.run.smithi049.stderr: 91.1% -- replaced with /var/log/ceph/8c050cea-2724-11ee-9b34-001a4aab830c/ceph-mon.a.log.gz 2023-07-20T17:51:35.573 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-07-20T17:51:35.574 INFO:teuthology.orchestra.run.smithi049.stderr:real 0m0.381s 2023-07-20T17:51:35.574 INFO:teuthology.orchestra.run.smithi049.stderr:user 0m1.172s 2023-07-20T17:51:35.574 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m0.064s 2023-07-20T17:51:35.575 INFO:tasks.cephadm:Archiving logs... 2023-07-20T17:51:35.576 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/remote/smithi049/log 2023-07-20T17:51:35.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T17:51:35.793 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/remote/smithi079/log 2023-07-20T17:51:35.794 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T17:51:35.928 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/remote/smithi097/log 2023-07-20T17:51:35.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T17:51:36.047 INFO:tasks.cephadm:Removing cluster... 2023-07-20T17:51:36.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8c050cea-2724-11ee-9b34-001a4aab830c --force 2023-07-20T17:51:36.339 INFO:teuthology.orchestra.run.smithi049.stdout:Deleting cluster with fsid: 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:51:36.709 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8c050cea-2724-11ee-9b34-001a4aab830c --force 2023-07-20T17:51:37.006 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:51:37.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8c050cea-2724-11ee-9b34-001a4aab830c --force 2023-07-20T17:51:37.758 INFO:teuthology.orchestra.run.smithi097.stdout:Deleting cluster with fsid: 8c050cea-2724-11ee-9b34-001a4aab830c 2023-07-20T17:51:38.211 INFO:tasks.cephadm:Removing cephadm ... 2023-07-20T17:51:38.212 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T17:51:38.236 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T17:51:38.256 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T17:51:38.274 INFO:tasks.cephadm:Teardown complete 2023-07-20T17:51:38.275 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' 2023-07-20T17:51:38.355 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c9f35efce74b40bd98105c616b76f6d2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' 2023-07-20T17:51:38.361 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-20T17:51:38.370 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-20T17:51:38.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-20T17:51:38.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T17:51:38.398 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T17:51:38.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T17:51:38.511 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-20T17:51:38.522 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-20T17:51:38.522 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T17:51:38.525 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T17:51:38.528 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T17:51:38.542 INFO:teuthology.orchestra.run.smithi049.stderr:bash: ntpq: command not found 2023-07-20T17:51:38.543 INFO:teuthology.orchestra.run.smithi079.stderr:bash: ntpq: command not found 2023-07-20T17:51:38.544 INFO:teuthology.orchestra.run.smithi097.stderr:bash: ntpq: command not found 2023-07-20T17:51:38.546 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T17:51:38.547 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-07-20T17:51:38.547 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +1989us[+1989us] +/- 107ms 2023-07-20T17:51:38.547 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv02.front.sepia.ceph.com 3 7 377 91 -255us[ -189us] +/- 85ms 2023-07-20T17:51:38.547 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -1473us[-1473us] +/- 104ms 2023-07-20T17:51:38.547 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:51:38.549 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T17:51:38.549 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2023-07-20T17:51:38.550 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 +1956us[+2020us] +/- 107ms 2023-07-20T17:51:38.550 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -337us[ -272us] +/- 86ms 2023-07-20T17:51:38.550 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 -1526us[-1526us] +/- 104ms 2023-07-20T17:51:38.551 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:51:38.552 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T17:51:38.552 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2023-07-20T17:51:38.552 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +1959us[+1959us] +/- 107ms 2023-07-20T17:51:38.552 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -314us[ -301us] +/- 86ms 2023-07-20T17:51:38.553 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -1507us[-1493us] +/- 104ms 2023-07-20T17:51:38.553 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T17:51:38.553 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-20T17:51:38.562 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-20T17:51:38.563 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-20T17:51:38.571 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T17:51:38.691 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T17:51:38.791 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T17:51:38.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T17:51:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:51:38.923 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T17:51:38.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:51:38.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T17:51:38.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:51:38.989 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-20T17:51:39.003 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-20T17:51:39.016 INFO:teuthology.task.internal:Duration was 2024.676055 seconds 2023-07-20T17:51:39.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-20T17:51:39.026 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-20T17:51:39.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T17:51:39.029 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T17:51:39.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T17:51:39.075 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T17:51:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T17:51:39.079 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T17:51:39.329 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-20T17:51:39.330 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2023-07-20T17:51:39.330 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T17:51:39.353 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2023-07-20T17:51:39.354 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T17:51:39.377 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2023-07-20T17:51:39.377 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T17:51:39.398 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-20T17:51:39.399 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T17:51:39.401 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T17:51:39.419 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T17:51:39.485 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-20T17:51:39.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T17:51:39.488 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T17:51:39.490 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T17:51:39.594 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-20T17:51:39.606 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-20T17:51:39.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T17:51:39.635 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T17:51:39.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T17:51:39.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-20T17:51:39.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T17:51:39.702 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T17:51:39.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T17:51:39.726 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2023-07-20T17:51:39.728 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2023-07-20T17:51:39.755 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2023-07-20T17:51:39.876 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T17:51:39.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:51:39.894 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T17:51:39.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:51:39.934 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T17:51:39.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T17:51:39.950 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-20T17:51:39.961 INFO:teuthology.task.internal:Transferring archived files... 2023-07-20T17:51:39.962 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/remote/smithi049 2023-07-20T17:51:39.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T17:51:40.006 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/remote/smithi079 2023-07-20T17:51:40.006 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T17:51:40.043 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/remote/smithi097 2023-07-20T17:51:40.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T17:51:40.082 INFO:teuthology.task.internal:Removing archive directory... 2023-07-20T17:51:40.082 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T17:51:40.085 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T17:51:40.087 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T17:51:40.132 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-20T17:51:40.144 INFO:teuthology.task.internal:Not uploading archives. 2023-07-20T17:51:40.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-20T17:51:40.154 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-20T17:51:40.154 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T17:51:40.156 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T17:51:40.159 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T17:51:40.171 INFO:teuthology.orchestra.run.smithi049.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 17:51 /home/ubuntu/cephtest 2023-07-20T17:51:40.172 INFO:teuthology.orchestra.run.smithi079.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 17:51 /home/ubuntu/cephtest 2023-07-20T17:51:40.189 INFO:teuthology.orchestra.run.smithi097.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 17:51 /home/ubuntu/cephtest 2023-07-20T17:51:40.190 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-20T17:51:40.203 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-20T17:51:40.224 INFO:teuthology.nuke:Checking targets against current locks 2023-07-20T17:51:40.253 DEBUG:teuthology.nuke:shortname: smithi049 2023-07-20T17:51:40.254 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T17:51:40.279 DEBUG:teuthology.nuke:shortname: smithi079 2023-07-20T17:51:40.279 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T17:51:40.284 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 17:11:23.773279', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T17:51:40.310 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 17:11:23.769981', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T17:51:40.311 DEBUG:teuthology.nuke:shortname: smithi097 2023-07-20T17:51:40.311 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T17:51:40.337 INFO:teuthology.orchestra.console:Power off smithi049 2023-07-20T17:51:40.339 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T17:51:40.353 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 17:11:23.772085', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T17:51:40.364 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T17:51:40.391 INFO:teuthology.orchestra.console:Power off smithi079 2023-07-20T17:51:40.392 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T17:51:40.417 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T17:51:40.440 INFO:teuthology.orchestra.console:Power off smithi097 2023-07-20T17:51:40.441 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T17:51:40.466 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T17:51:44.410 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T17:51:44.453 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T17:51:44.490 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T17:51:48.528 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T17:51:48.566 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T17:51:48.603 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T17:51:48.745 INFO:teuthology.orchestra.console:Power off for smithi049 completed 2023-07-20T17:51:52.684 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T17:51:52.716 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T17:51:52.896 INFO:teuthology.orchestra.console:Power off for smithi079 completed 2023-07-20T17:51:52.930 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2023-07-20T17:51:53.042 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} duration: 2024.6760551929474 failure_reason: 'Command failed on smithi049 with status 2: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c9f35efce74b40bd98105c616b76f6d2 status: fail success: false 2023-07-20T17:51:53.043 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T17:51:53.129 INFO:teuthology.run:FAIL