2023-07-20T01:21:32.428 INFO:root:teuthology version: 0.0.1.dev153+g79491cb 2023-07-20T01:21:32.429 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} --name pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251 --verbose --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251/orig.config.yaml 2023-07-20T01:21:32.502 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T01:21:32.600 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251 branch: wip-pdonnell-testing-20230719.201220 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} email: pdonnell@redhat.com first_in_suite: false job_id: '7344251' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230719.201220 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.dir_split install: ceph: flavor: default sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230719.201220 sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 6483 sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 sleep_before_teardown: 0 subset: 383/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230719.201220 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi003.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi086.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/misc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 79491cb69dd37a1eafb93b3471998174bdfa2fd9 timestamp: 2023-07-20_00:39:06 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-20T01:21:32.601 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa; will attempt to use it 2023-07-20T01:21:32.602 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks 2023-07-20T01:21:32.602 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-20T01:21:32.604 INFO:teuthology.task.internal:Checking packages... 2023-07-20T01:21:32.632 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '1b0b2eb005a7c51d97a788b9900d511e1394b537' 2023-07-20T01:21:32.632 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-20T01:21:32.633 INFO:teuthology.packaging:ref: None 2023-07-20T01:21:32.633 INFO:teuthology.packaging:tag: None 2023-07-20T01:21:32.633 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230719.201220 2023-07-20T01:21:32.633 INFO:teuthology.packaging:sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:21:32.634 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230719.201220 2023-07-20T01:21:32.975 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5040.g1b0b2eb0 2023-07-20T01:21:32.975 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-20T01:21:32.996 INFO:teuthology.task.internal:no buildpackages task found 2023-07-20T01:21:32.997 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-20T01:21:33.058 INFO:teuthology.task.internal:Saving configuration 2023-07-20T01:21:33.080 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-20T01:21:33.111 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T01:21:33.136 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 01:16:48.200735', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T01:21:33.162 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 01:16:48.197624', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T01:21:33.185 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 01:16:48.199280', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T01:21:33.186 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-20T01:21:33.194 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-20T01:21:33.194 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-20T01:21:33.194 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-20T01:21:33.195 INFO:teuthology.run_tasks:Running task console_log... 2023-07-20T01:21:33.344 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbf916a63a0>, signals=[15]) 2023-07-20T01:21:33.345 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-20T01:21:33.353 INFO:teuthology.task.internal:Opening connections... 2023-07-20T01:21:33.354 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2023-07-20T01:21:33.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:21:33.425 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2023-07-20T01:21:33.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:21:33.492 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2023-07-20T01:21:33.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:21:33.560 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-20T01:21:33.568 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2023-07-20T01:21:33.589 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2023-07-20T01:21:33.590 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2023-07-20T01:21:33.649 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T01:21:33.649 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T01:21:33.650 INFO:teuthology.orchestra.run.smithi003.stdout:ID="rhel" 2023-07-20T01:21:33.650 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="fedora" 2023-07-20T01:21:33.650 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="8.6" 2023-07-20T01:21:33.650 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el8" 2023-07-20T01:21:33.650 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T01:21:33.650 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2023-07-20T01:21:33.650 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T01:21:33.651 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T01:21:33.651 INFO:teuthology.orchestra.run.smithi003.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T01:21:33.651 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T01:21:33.651 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:21:33.651 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T01:21:33.651 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T01:21:33.651 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T01:21:33.651 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T01:21:33.652 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2023-07-20T01:21:33.675 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2023-07-20T01:21:33.695 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2023-07-20T01:21:33.696 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2023-07-20T01:21:33.755 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T01:21:33.755 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T01:21:33.755 INFO:teuthology.orchestra.run.smithi043.stdout:ID="rhel" 2023-07-20T01:21:33.756 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="fedora" 2023-07-20T01:21:33.756 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="8.6" 2023-07-20T01:21:33.756 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el8" 2023-07-20T01:21:33.756 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T01:21:33.756 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31" 2023-07-20T01:21:33.756 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T01:21:33.756 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T01:21:33.757 INFO:teuthology.orchestra.run.smithi043.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T01:21:33.757 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T01:21:33.757 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:21:33.757 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T01:21:33.757 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T01:21:33.757 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T01:21:33.757 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T01:21:33.758 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2023-07-20T01:21:33.783 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2023-07-20T01:21:33.803 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2023-07-20T01:21:33.804 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2023-07-20T01:21:33.862 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T01:21:33.862 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T01:21:33.863 INFO:teuthology.orchestra.run.smithi086.stdout:ID="rhel" 2023-07-20T01:21:33.863 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="fedora" 2023-07-20T01:21:33.863 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8.6" 2023-07-20T01:21:33.863 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8" 2023-07-20T01:21:33.863 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T01:21:33.863 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2023-07-20T01:21:33.863 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T01:21:33.864 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T01:21:33.864 INFO:teuthology.orchestra.run.smithi086.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T01:21:33.864 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T01:21:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:21:33.864 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T01:21:33.864 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T01:21:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T01:21:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T01:21:33.865 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2023-07-20T01:21:33.891 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-20T01:21:33.904 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-20T01:21:33.946 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-20T01:21:33.947 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2023-07-20T01:21:33.949 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2023-07-20T01:21:33.951 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2023-07-20T01:21:33.970 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-20T01:21:33.981 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-20T01:21:33.981 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2023-07-20T01:21:34.009 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2023-07-20T01:21:34.011 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2023-07-20T01:21:34.039 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-20T01:21:34.130 INFO:teuthology.run_tasks:Running task kernel... 2023-07-20T01:21:34.144 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-07-20T01:21:34.145 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-07-20T01:21:34.145 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-07-20T01:21:34.145 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T01:21:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:21:34.165 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-07-20T01:21:34.225 INFO:teuthology.orchestra.run.smithi003.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-20T01:21:34.226 INFO:teuthology.task.kernel:Running kernel on smithi003: 4.18.0-372.9.1.el8.x86_64 2023-07-20T01:21:34.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2023-07-20T01:21:35.384 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-07-20T01:21:35.385 INFO:teuthology.orchestra.run.smithi003.stdout:Unable to read consumer identity 2023-07-20T01:21:36.811 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-07-20T01:21:52.209 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 16 MB 00:15 2023-07-20T01:21:55.809 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 608 kB/s | 24 kB 00:00 2023-07-20T01:21:56.728 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-20T01:21:56.787 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-07-20T01:21:56.789 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-07-20T01:21:56.789 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-07-20T01:21:56.923 DEBUG:teuthology.orchestra.run.smithi003:> echo no | sudo yum reinstall kernel || true 2023-07-20T01:21:57.278 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-07-20T01:21:57.279 INFO:teuthology.orchestra.run.smithi003.stdout:Unable to read consumer identity 2023-07-20T01:21:57.559 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:02 ago on Thu 20 Jul 2023 01:21:55 AM UTC. 2023-07-20T01:21:57.603 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T01:21:57.645 INFO:teuthology.orchestra.run.smithi003.stderr:Error: No packages marked for reinstall. 2023-07-20T01:21:57.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum reinstall -y kernel || true 2023-07-20T01:21:58.042 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-07-20T01:21:58.042 INFO:teuthology.orchestra.run.smithi003.stdout:Unable to read consumer identity 2023-07-20T01:21:58.313 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:03 ago on Thu 20 Jul 2023 01:21:55 AM UTC. 2023-07-20T01:21:58.357 INFO:teuthology.orchestra.run.smithi003.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T01:21:58.386 INFO:teuthology.orchestra.run.smithi003.stderr:Error: No packages marked for reinstall. 2023-07-20T01:21:58.433 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2023-07-20T01:21:58.516 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-20T01:21:58.516 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-20T01:21:58.517 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-20T01:21:58.517 DEBUG:teuthology.parallel:result is None 2023-07-20T01:21:58.517 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-20T01:21:58.524 INFO:teuthology.task.internal:Creating test directory... 2023-07-20T01:21:58.525 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T01:21:58.527 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T01:21:58.529 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T01:21:58.551 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-20T01:21:58.574 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-20T01:21:58.607 INFO:teuthology.task.internal:Creating archive directory... 2023-07-20T01:21:58.608 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T01:21:58.610 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T01:21:58.612 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T01:21:58.659 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-20T01:21:58.676 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-20T01:21:58.676 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T01:21:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:21:58.710 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T01:21:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:21:58.730 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T01:21:58.732 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T01:21:58.752 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T01:21:58.758 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:21:58.774 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:21:58.785 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:21:58.799 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:21:58.816 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:21:58.834 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:21:58.835 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-20T01:21:58.847 INFO:teuthology.task.internal:Configuring sudo... 2023-07-20T01:21:58.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T01:21:58.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T01:21:58.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T01:21:58.880 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-20T01:21:58.891 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-20T01:21:58.891 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T01:21:58.916 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T01:21:58.922 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T01:21:58.939 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:21:59.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:21:59.066 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:21:59.153 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:21:59.191 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:21:59.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T01:21:59.259 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:21:59.309 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:21:59.367 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:21:59.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:21:59.458 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2023-07-20T01:21:59.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T01:21:59.528 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:21:59.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:21:59.627 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:21:59.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:21:59.755 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-07-20T01:21:59.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T01:21:59.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2023-07-20T01:21:59.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2023-07-20T01:21:59.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2023-07-20T01:21:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:21:59.890 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:21:59.928 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:22:00.381 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-20T01:22:00.388 INFO:teuthology.task.internal:Starting timer... 2023-07-20T01:22:00.388 INFO:teuthology.run_tasks:Running task pcp... 2023-07-20T01:22:00.456 INFO:teuthology.run_tasks:Running task selinux... 2023-07-20T01:22:00.496 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-07-20T01:22:00.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2023-07-20T01:22:00.563 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2023-07-20T01:22:00.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo service auditd rotate 2023-07-20T01:22:00.630 INFO:teuthology.orchestra.run.smithi043.stdout:Rotating logs: 2023-07-20T01:22:00.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2023-07-20T01:22:00.682 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2023-07-20T01:22:00.684 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-20T01:22:00.684 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2023-07-20T01:22:00.716 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2023-07-20T01:22:00.716 DEBUG:teuthology.orchestra.run.smithi043:> /usr/sbin/getenforce 2023-07-20T01:22:00.744 INFO:teuthology.orchestra.run.smithi043.stdout:Permissive 2023-07-20T01:22:00.745 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2023-07-20T01:22:00.764 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2023-07-20T01:22:00.765 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi043.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2023-07-20T01:22:00.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:22:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:22:00.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:22:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:22:00.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:22:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:22:00.851 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-20T01:22:00.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2023-07-20T01:22:00.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo /usr/sbin/setenforce permissive 2023-07-20T01:22:00.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2023-07-20T01:22:00.932 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-20T01:22:00.942 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-20T01:22:00.947 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-20T01:22:01.086 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-20T01:22:01.104 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-20T01:22:01.105 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi043.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-20T01:30:56.299 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi043.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2023-07-20T01:30:56.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-07-20T01:30:56.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:30:56.373 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-07-20T01:30:56.442 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-07-20T01:30:56.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2023-07-20T01:30:56.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:30:56.514 DEBUG:teuthology.orchestra.run.smithi043:> true 2023-07-20T01:30:56.588 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2023-07-20T01:30:56.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-07-20T01:30:56.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:30:56.655 DEBUG:teuthology.orchestra.run.smithi086:> true 2023-07-20T01:30:56.680 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-07-20T01:30:56.680 INFO:teuthology.run_tasks:Running task clock... 2023-07-20T01:30:56.691 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-20T01:30:56.691 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T01:30:56.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:30:56.694 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T01:30:56.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:30:56.697 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T01:30:56.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:30:56.726 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T01:30:56.732 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T01:30:56.743 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T01:30:56.749 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T01:30:56.752 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T01:30:56.767 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T01:30:56.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2023-07-20T01:30:56.821 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2023-07-20T01:30:56.837 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T01:30:56.838 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: ntpd: command not found 2023-07-20T01:30:56.841 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2023-07-20T01:30:56.851 INFO:teuthology.orchestra.run.smithi043.stdout:506 Cannot talk to daemon 2023-07-20T01:30:56.852 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T01:30:56.852 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2023-07-20T01:30:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T01:30:56.867 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T01:30:56.880 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T01:30:56.883 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T01:30:56.926 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2023-07-20T01:30:56.929 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:30:56.929 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-07-20T01:30:56.929 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.929 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.929 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.930 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.953 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2023-07-20T01:30:56.956 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:30:56.956 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2023-07-20T01:30:56.956 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.956 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.957 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.957 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:bash: ntpq: command not found 2023-07-20T01:30:56.965 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:30:56.965 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2023-07-20T01:30:56.965 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.965 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.965 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.966 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:56.966 INFO:teuthology.run_tasks:Running task pexec... 2023-07-20T01:30:56.976 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-20T01:30:56.977 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi003.front.sepia.ceph.com 2023-07-20T01:30:56.977 DEBUG:teuthology.orchestra.run.smithi003:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T01:30:56.978 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi043.front.sepia.ceph.com 2023-07-20T01:30:56.978 DEBUG:teuthology.orchestra.run.smithi043:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T01:30:56.979 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi086.front.sepia.ceph.com 2023-07-20T01:30:56.979 DEBUG:teuthology.orchestra.run.smithi086:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T01:30:57.368 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-07-20T01:30:57.369 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-07-20T01:30:57.402 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2023-07-20T01:30:58.360 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:53 ago on Thu 20 Jul 2023 01:29:04 AM UTC. 2023-07-20T01:30:58.379 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:35 ago on Thu 20 Jul 2023 01:29:22 AM UTC. 2023-07-20T01:30:58.420 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:38 ago on Thu 20 Jul 2023 01:29:19 AM UTC. 2023-07-20T01:31:00.119 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-07-20T01:31:00.120 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-07-20T01:31:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-07-20T01:31:00.120 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-07-20T01:31:00.120 INFO:teuthology.orchestra.run.smithi086.stdout:Resetting modules: 2023-07-20T01:31:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools 2023-07-20T01:31:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:31:00.121 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-07-20T01:31:00.121 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-07-20T01:31:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:31:00.390 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-07-20T01:31:00.391 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-07-20T01:31:00.391 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-07-20T01:31:00.391 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-07-20T01:31:00.391 INFO:teuthology.orchestra.run.smithi003.stdout:Resetting modules: 2023-07-20T01:31:00.392 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools 2023-07-20T01:31:00.392 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:31:00.392 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-07-20T01:31:00.392 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-07-20T01:31:00.392 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:31:00.417 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2023-07-20T01:31:00.418 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2023-07-20T01:31:00.418 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2023-07-20T01:31:00.419 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2023-07-20T01:31:00.419 INFO:teuthology.orchestra.run.smithi043.stdout:Resetting modules: 2023-07-20T01:31:00.419 INFO:teuthology.orchestra.run.smithi043.stdout: container-tools 2023-07-20T01:31:00.419 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:31:00.419 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2023-07-20T01:31:00.419 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2023-07-20T01:31:00.419 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:31:01.084 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:56 ago on Thu 20 Jul 2023 01:29:04 AM UTC. 2023-07-20T01:31:01.403 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:38 ago on Thu 20 Jul 2023 01:29:22 AM UTC. 2023-07-20T01:31:01.430 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:42 ago on Thu 20 Jul 2023 01:29:19 AM UTC. 2023-07-20T01:31:03.429 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-07-20T01:31:03.908 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-07-20T01:31:04.364 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-07-20T01:31:04.431 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2023-07-20T01:31:04.819 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:00 ago on Thu 20 Jul 2023 01:29:04 AM UTC. 2023-07-20T01:31:04.858 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-07-20T01:31:04.917 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2023-07-20T01:31:05.742 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:43 ago on Thu 20 Jul 2023 01:29:22 AM UTC. 2023-07-20T01:31:05.790 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:46 ago on Thu 20 Jul 2023 01:29:19 AM UTC. 2023-07-20T01:31:06.533 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-07-20T01:31:06.623 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================================ 2023-07-20T01:31:06.623 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-07-20T01:31:06.623 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================================ 2023-07-20T01:31:06.624 INFO:teuthology.orchestra.run.smithi086.stdout:Installing group/module packages: 2023-07-20T01:31:06.624 INFO:teuthology.orchestra.run.smithi086.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T01:31:06.624 INFO:teuthology.orchestra.run.smithi086.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T01:31:06.624 INFO:teuthology.orchestra.run.smithi086.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T01:31:06.624 INFO:teuthology.orchestra.run.smithi086.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T01:31:06.624 INFO:teuthology.orchestra.run.smithi086.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T01:31:06.625 INFO:teuthology.orchestra.run.smithi086.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T01:31:06.625 INFO:teuthology.orchestra.run.smithi086.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T01:31:06.625 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-07-20T01:31:06.625 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T01:31:06.625 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T01:31:06.625 INFO:teuthology.orchestra.run.smithi086.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T01:31:06.626 INFO:teuthology.orchestra.run.smithi086.stdout:Installing module profiles: 2023-07-20T01:31:06.626 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools/common 2023-07-20T01:31:06.626 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:31:06.626 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling module streams: 2023-07-20T01:31:06.626 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools rhel8 2023-07-20T01:31:06.626 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:31:06.626 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-07-20T01:31:06.627 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================================ 2023-07-20T01:31:06.627 INFO:teuthology.orchestra.run.smithi086.stdout:Install 10 Packages 2023-07-20T01:31:06.627 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:31:06.628 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 18 M 2023-07-20T01:31:06.628 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 65 M 2023-07-20T01:31:06.628 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-07-20T01:31:06.694 INFO:teuthology.orchestra.run.smithi086.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.3 MB/s | 94 kB 00:00 2023-07-20T01:31:06.739 INFO:teuthology.orchestra.run.smithi086.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 218 kB/s | 25 kB 00:00 2023-07-20T01:31:06.761 INFO:teuthology.orchestra.run.smithi086.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 301 kB/s | 41 kB 00:00 2023-07-20T01:31:06.778 INFO:teuthology.orchestra.run.smithi086.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 589 kB/s | 49 kB 00:00 2023-07-20T01:31:06.812 INFO:teuthology.orchestra.run.smithi086.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 7.1 MB/s | 524 kB 00:00 2023-07-20T01:31:06.837 INFO:teuthology.orchestra.run.smithi086.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-07-20T01:31:06.862 INFO:teuthology.orchestra.run.smithi086.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-07-20T01:31:06.946 INFO:teuthology.orchestra.run.smithi086.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 26 MB/s | 2.2 MB 00:00 2023-07-20T01:31:07.054 INFO:teuthology.orchestra.run.smithi086.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 32 MB/s | 7.0 MB 00:00 2023-07-20T01:31:07.138 INFO:teuthology.orchestra.run.smithi086.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 21 MB/s | 8.0 MB 00:00 2023-07-20T01:31:07.138 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:31:07.139 INFO:teuthology.orchestra.run.smithi086.stdout:Total 36 MB/s | 18 MB 00:00 2023-07-20T01:31:07.339 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-07-20T01:31:07.357 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-07-20T01:31:07.357 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-07-20T01:31:07.538 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-07-20T01:31:07.542 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-07-20T01:31:07.644 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-07-20T01:31:07.748 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================================ 2023-07-20T01:31:07.748 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-07-20T01:31:07.748 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================================ 2023-07-20T01:31:07.748 INFO:teuthology.orchestra.run.smithi003.stdout:Installing group/module packages: 2023-07-20T01:31:07.749 INFO:teuthology.orchestra.run.smithi003.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T01:31:07.749 INFO:teuthology.orchestra.run.smithi003.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T01:31:07.749 INFO:teuthology.orchestra.run.smithi003.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T01:31:07.749 INFO:teuthology.orchestra.run.smithi003.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T01:31:07.749 INFO:teuthology.orchestra.run.smithi003.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T01:31:07.749 INFO:teuthology.orchestra.run.smithi003.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T01:31:07.749 INFO:teuthology.orchestra.run.smithi003.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T01:31:07.750 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-07-20T01:31:07.750 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T01:31:07.750 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T01:31:07.750 INFO:teuthology.orchestra.run.smithi003.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T01:31:07.750 INFO:teuthology.orchestra.run.smithi003.stdout:Installing module profiles: 2023-07-20T01:31:07.750 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools/common 2023-07-20T01:31:07.750 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:31:07.750 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling module streams: 2023-07-20T01:31:07.751 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools rhel8 2023-07-20T01:31:07.751 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:31:07.751 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-07-20T01:31:07.751 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================================ 2023-07-20T01:31:07.751 INFO:teuthology.orchestra.run.smithi003.stdout:Install 10 Packages 2023-07-20T01:31:07.751 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:31:07.752 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 18 M 2023-07-20T01:31:07.752 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 65 M 2023-07-20T01:31:07.752 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-07-20T01:31:07.775 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2023-07-20T01:31:07.827 INFO:teuthology.orchestra.run.smithi003.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2023-07-20T01:31:07.876 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================================ 2023-07-20T01:31:07.876 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2023-07-20T01:31:07.876 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================================ 2023-07-20T01:31:07.876 INFO:teuthology.orchestra.run.smithi043.stdout:Installing group/module packages: 2023-07-20T01:31:07.876 INFO:teuthology.orchestra.run.smithi043.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T01:31:07.877 INFO:teuthology.orchestra.run.smithi043.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T01:31:07.877 INFO:teuthology.orchestra.run.smithi043.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T01:31:07.877 INFO:teuthology.orchestra.run.smithi043.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T01:31:07.877 INFO:teuthology.orchestra.run.smithi043.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T01:31:07.877 INFO:teuthology.orchestra.run.smithi043.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T01:31:07.877 INFO:teuthology.orchestra.run.smithi043.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T01:31:07.877 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2023-07-20T01:31:07.878 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T01:31:07.878 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T01:31:07.878 INFO:teuthology.orchestra.run.smithi043.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T01:31:07.878 INFO:teuthology.orchestra.run.smithi043.stdout:Installing module profiles: 2023-07-20T01:31:07.878 INFO:teuthology.orchestra.run.smithi043.stdout: container-tools/common 2023-07-20T01:31:07.878 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:31:07.878 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling module streams: 2023-07-20T01:31:07.878 INFO:teuthology.orchestra.run.smithi043.stdout: container-tools rhel8 2023-07-20T01:31:07.879 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:31:07.879 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2023-07-20T01:31:07.879 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================================ 2023-07-20T01:31:07.879 INFO:teuthology.orchestra.run.smithi043.stdout:Install 10 Packages 2023-07-20T01:31:07.879 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:31:07.880 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 18 M 2023-07-20T01:31:07.880 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 65 M 2023-07-20T01:31:07.880 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2023-07-20T01:31:07.886 INFO:teuthology.orchestra.run.smithi003.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 184 kB/s | 25 kB 00:00 2023-07-20T01:31:07.911 INFO:teuthology.orchestra.run.smithi003.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 254 kB/s | 41 kB 00:00 2023-07-20T01:31:07.937 INFO:teuthology.orchestra.run.smithi003.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 454 kB/s | 49 kB 00:00 2023-07-20T01:31:07.951 INFO:teuthology.orchestra.run.smithi043.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.3 MB/s | 94 kB 00:00 2023-07-20T01:31:07.970 INFO:teuthology.orchestra.run.smithi003.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.1 MB/s | 524 kB 00:00 2023-07-20T01:31:07.980 INFO:teuthology.orchestra.run.smithi043.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 253 kB/s | 25 kB 00:00 2023-07-20T01:31:07.995 INFO:teuthology.orchestra.run.smithi003.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-07-20T01:31:08.002 INFO:teuthology.orchestra.run.smithi043.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 331 kB/s | 41 kB 00:00 2023-07-20T01:31:08.012 INFO:teuthology.orchestra.run.smithi003.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 5.0 MB/s | 213 kB 00:00 2023-07-20T01:31:08.027 INFO:teuthology.orchestra.run.smithi043.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 653 kB/s | 49 kB 00:00 2023-07-20T01:31:08.061 INFO:teuthology.orchestra.run.smithi043.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.1 MB/s | 524 kB 00:00 2023-07-20T01:31:08.078 INFO:teuthology.orchestra.run.smithi043.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.9 MB/s | 149 kB 00:00 2023-07-20T01:31:08.081 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-07-20T01:31:08.103 INFO:teuthology.orchestra.run.smithi043.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 5.0 MB/s | 213 kB 00:00 2023-07-20T01:31:08.113 INFO:teuthology.orchestra.run.smithi003.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 22 MB/s | 2.2 MB 00:00 2023-07-20T01:31:08.195 INFO:teuthology.orchestra.run.smithi043.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-07-20T01:31:08.238 INFO:teuthology.orchestra.run.smithi003.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 29 MB/s | 7.0 MB 00:00 2023-07-20T01:31:08.265 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T01:31:08.320 INFO:teuthology.orchestra.run.smithi043.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 29 MB/s | 7.0 MB 00:00 2023-07-20T01:31:08.321 INFO:teuthology.orchestra.run.smithi003.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 19 MB/s | 8.0 MB 00:00 2023-07-20T01:31:08.322 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:31:08.322 INFO:teuthology.orchestra.run.smithi003.stdout:Total 32 MB/s | 18 MB 00:00 2023-07-20T01:31:08.423 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:31:08.442 INFO:teuthology.orchestra.run.smithi043.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 18 MB/s | 8.0 MB 00:00 2023-07-20T01:31:08.443 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:31:08.443 INFO:teuthology.orchestra.run.smithi043.stdout:Total 32 MB/s | 18 MB 00:00 2023-07-20T01:31:08.529 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-07-20T01:31:08.550 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-07-20T01:31:08.550 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-07-20T01:31:08.650 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2023-07-20T01:31:08.651 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T01:31:08.670 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2023-07-20T01:31:08.671 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2023-07-20T01:31:08.744 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-07-20T01:31:08.749 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-07-20T01:31:08.868 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2023-07-20T01:31:08.872 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2023-07-20T01:31:08.878 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T01:31:09.251 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T01:31:09.269 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-07-20T01:31:09.379 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2023-07-20T01:31:09.444 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T01:31:09.570 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T01:31:09.622 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:31:09.759 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:31:09.795 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T01:31:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T01:31:09.991 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T01:31:10.014 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T01:31:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T01:31:10.337 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T01:31:10.509 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T01:31:10.887 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T01:31:11.074 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T01:31:11.137 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T01:31:11.230 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T01:31:11.272 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T01:31:11.432 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:31:11.699 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:31:11.699 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T01:31:11.699 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:31:11.700 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T01:31:11.700 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T01:31:11.700 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T01:31:11.700 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T01:31:11.700 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T01:31:11.700 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T01:31:11.700 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T01:31:12.058 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T01:31:12.186 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T01:31:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T01:31:12.420 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T01:31:12.454 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T01:31:12.590 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T01:31:12.622 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:31:12.758 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:31:12.883 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T01:31:12.883 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:08 ago on Thu 20 Jul 2023 01:29:04 AM UTC. 2023-07-20T01:31:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:31:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T01:31:12.890 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:31:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T01:31:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T01:31:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T01:31:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T01:31:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T01:31:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T01:31:12.891 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T01:31:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:31:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T01:31:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:31:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T01:31:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T01:31:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T01:31:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T01:31:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T01:31:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T01:31:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T01:31:14.176 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T01:31:14.177 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:51 ago on Thu 20 Jul 2023 01:29:22 AM UTC. 2023-07-20T01:31:14.208 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T01:31:14.209 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:54 ago on Thu 20 Jul 2023 01:29:19 AM UTC. 2023-07-20T01:31:16.177 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-07-20T01:31:16.369 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:31:16.370 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-07-20T01:31:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:31:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:16.370 INFO:teuthology.orchestra.run.smithi086.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T01:31:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T01:31:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T01:31:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:31:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T01:31:16.371 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:31:16.372 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-07-20T01:31:16.515 DEBUG:teuthology.parallel:result is None 2023-07-20T01:31:17.325 INFO:teuthology.orchestra.run.smithi003.stdout:Installed products updated. 2023-07-20T01:31:17.366 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2023-07-20T01:31:17.567 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:31:17.568 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-07-20T01:31:17.568 INFO:teuthology.orchestra.run.smithi003.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:17.568 INFO:teuthology.orchestra.run.smithi003.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:31:17.568 INFO:teuthology.orchestra.run.smithi003.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:17.568 INFO:teuthology.orchestra.run.smithi003.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T01:31:17.568 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T01:31:17.569 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T01:31:17.569 INFO:teuthology.orchestra.run.smithi003.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:17.569 INFO:teuthology.orchestra.run.smithi003.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:17.569 INFO:teuthology.orchestra.run.smithi003.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:31:17.569 INFO:teuthology.orchestra.run.smithi003.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T01:31:17.569 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:31:17.569 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-07-20T01:31:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:31:17.578 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2023-07-20T01:31:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:17.578 INFO:teuthology.orchestra.run.smithi043.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:31:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T01:31:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T01:31:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T01:31:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:31:17.579 INFO:teuthology.orchestra.run.smithi043.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:31:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T01:31:17.580 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:31:17.580 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2023-07-20T01:31:17.715 DEBUG:teuthology.parallel:result is None 2023-07-20T01:31:17.722 DEBUG:teuthology.parallel:result is None 2023-07-20T01:31:17.723 INFO:teuthology.run_tasks:Running task install... 2023-07-20T01:31:17.732 DEBUG:teuthology.task.install:project ceph 2023-07-20T01:31:17.732 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'}} 2023-07-20T01:31:17.733 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'} 2023-07-20T01:31:17.733 INFO:teuthology.task.install:Using flavor: default 2023-07-20T01:31:17.739 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-20T01:31:17.739 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-20T01:31:17.740 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T01:31:17.740 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:31:17.742 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T01:31:17.743 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:31:17.744 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T01:31:17.744 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:31:18.068 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T01:31:18.068 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T01:31:18.081 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T01:31:18.081 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T01:31:18.085 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T01:31:18.085 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T01:31:18.337 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T01:31:18.337 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2023-07-20T01:31:18.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T01:31:18.340 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T01:31:18.340 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-07-20T01:31:18.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T01:31:18.353 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T01:31:18.353 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:31:18.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T01:31:18.372 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T01:31:18.373 DEBUG:teuthology.orchestra.run.smithi043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T01:31:18.375 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T01:31:18.375 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T01:31:18.384 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T01:31:18.384 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T01:31:18.444 DEBUG:teuthology.orchestra.run.smithi043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T01:31:18.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T01:31:18.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T01:31:18.525 DEBUG:teuthology.orchestra.run.smithi043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T01:31:18.529 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T01:31:18.535 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T01:31:18.594 INFO:teuthology.orchestra.run.smithi043.stdout:check_obsoletes = 1 2023-07-20T01:31:18.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum clean all 2023-07-20T01:31:18.597 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2023-07-20T01:31:18.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2023-07-20T01:31:18.602 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2023-07-20T01:31:18.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2023-07-20T01:31:18.972 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-07-20T01:31:18.992 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2023-07-20T01:31:18.997 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-07-20T01:31:19.653 INFO:teuthology.orchestra.run.smithi086.stdout:44 files removed 2023-07-20T01:31:19.668 INFO:teuthology.orchestra.run.smithi003.stdout:44 files removed 2023-07-20T01:31:19.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T01:31:19.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T01:31:19.725 INFO:teuthology.orchestra.run.smithi043.stdout:44 files removed 2023-07-20T01:31:19.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T01:31:20.029 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-07-20T01:31:20.070 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-07-20T01:31:20.130 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2023-07-20T01:31:22.398 INFO:teuthology.orchestra.run.smithi043.stdout:ceph packages for x86_64 68 kB/s | 81 kB 00:01 2023-07-20T01:31:22.889 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 50 kB/s | 81 kB 00:01 2023-07-20T01:31:22.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 44 kB/s | 81 kB 00:01 2023-07-20T01:31:22.998 INFO:teuthology.orchestra.run.smithi043.stdout:ceph noarch packages 17 kB/s | 9.4 kB 00:00 2023-07-20T01:31:23.315 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 24 kB/s | 9.4 kB 00:00 2023-07-20T01:31:23.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 25 kB/s | 9.4 kB 00:00 2023-07-20T01:31:23.424 INFO:teuthology.orchestra.run.smithi043.stdout:ceph source packages 3.7 kB/s | 1.5 kB 00:00 2023-07-20T01:31:23.608 INFO:teuthology.orchestra.run.smithi043.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-07-20T01:31:23.715 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 4.0 kB/s | 1.5 kB 00:00 2023-07-20T01:31:23.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 4.0 kB/s | 1.5 kB 00:00 2023-07-20T01:31:23.916 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-07-20T01:31:24.031 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-07-20T01:31:25.059 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-07-20T01:31:28.684 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 404 kB/s | 24 kB 00:00 2023-07-20T01:31:30.235 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 36 MB/s | 53 MB 00:01 2023-07-20T01:31:36.999 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 1.2 MB/s | 16 MB 00:13 2023-07-20T01:31:38.952 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 MB/s | 47 MB 00:00 2023-07-20T01:31:40.640 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 508 kB/s | 24 kB 00:00 2023-07-20T01:31:42.309 INFO:teuthology.orchestra.run.smithi003.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 MB/s | 47 MB 00:01 2023-07-20T01:31:50.636 INFO:teuthology.orchestra.run.smithi043.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:31:50.637 INFO:teuthology.orchestra.run.smithi043.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:31:50.654 INFO:teuthology.orchestra.run.smithi043.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T01:31:50.654 INFO:teuthology.orchestra.run.smithi043.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T01:31:50.655 INFO:teuthology.orchestra.run.smithi043.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T01:31:50.656 INFO:teuthology.orchestra.run.smithi043.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T01:31:50.658 INFO:teuthology.orchestra.run.smithi043.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T01:31:50.662 INFO:teuthology.orchestra.run.smithi043.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T01:31:50.663 INFO:teuthology.orchestra.run.smithi043.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T01:31:50.710 INFO:teuthology.orchestra.run.smithi003.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 59 MB/s | 53 MB 00:00 2023-07-20T01:31:50.810 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2023-07-20T01:31:50.823 INFO:teuthology.orchestra.run.smithi043.stdout:=========================================================================================================================== 2023-07-20T01:31:50.824 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2023-07-20T01:31:50.825 INFO:teuthology.orchestra.run.smithi043.stdout:=========================================================================================================================== 2023-07-20T01:31:50.825 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2023-07-20T01:31:50.825 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T01:31:50.826 INFO:teuthology.orchestra.run.smithi043.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T01:31:50.826 INFO:teuthology.orchestra.run.smithi043.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T01:31:50.826 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T01:31:50.826 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T01:31:50.827 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T01:31:50.827 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T01:31:50.827 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T01:31:50.828 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T01:31:50.828 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T01:31:50.828 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T01:31:50.828 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T01:31:50.829 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T01:31:50.829 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T01:31:50.829 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T01:31:50.830 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T01:31:50.830 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T01:31:50.831 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T01:31:50.831 INFO:teuthology.orchestra.run.smithi043.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T01:31:50.831 INFO:teuthology.orchestra.run.smithi043.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T01:31:50.831 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T01:31:50.831 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T01:31:50.831 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:31:50.831 INFO:teuthology.orchestra.run.smithi043.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T01:31:50.831 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T01:31:50.832 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T01:31:50.832 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T01:31:50.832 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T01:31:50.832 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T01:31:50.832 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T01:31:50.833 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T01:31:50.833 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T01:31:50.833 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T01:31:50.833 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T01:31:50.833 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T01:31:50.834 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T01:31:50.834 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T01:31:50.834 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2023-07-20T01:31:50.835 INFO:teuthology.orchestra.run.smithi043.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T01:31:50.835 INFO:teuthology.orchestra.run.smithi043.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T01:31:50.835 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T01:31:50.836 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2023-07-20T01:31:50.836 INFO:teuthology.orchestra.run.smithi043.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T01:31:50.836 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T01:31:50.837 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T01:31:50.837 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T01:31:50.837 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T01:31:50.837 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T01:31:50.838 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T01:31:50.838 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T01:31:50.838 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T01:31:50.839 INFO:teuthology.orchestra.run.smithi043.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T01:31:50.839 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T01:31:50.839 INFO:teuthology.orchestra.run.smithi043.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T01:31:50.840 INFO:teuthology.orchestra.run.smithi043.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T01:31:50.840 INFO:teuthology.orchestra.run.smithi043.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T01:31:50.840 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:31:50.841 INFO:teuthology.orchestra.run.smithi043.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T01:31:50.841 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T01:31:50.841 INFO:teuthology.orchestra.run.smithi043.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T01:31:50.841 INFO:teuthology.orchestra.run.smithi043.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T01:31:50.842 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T01:31:50.842 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T01:31:50.842 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T01:31:50.842 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T01:31:50.842 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T01:31:50.842 INFO:teuthology.orchestra.run.smithi043.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T01:31:50.842 INFO:teuthology.orchestra.run.smithi043.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T01:31:50.843 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T01:31:50.843 INFO:teuthology.orchestra.run.smithi043.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T01:31:50.843 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T01:31:50.843 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T01:31:50.843 INFO:teuthology.orchestra.run.smithi043.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T01:31:50.843 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T01:31:50.844 INFO:teuthology.orchestra.run.smithi043.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T01:31:50.844 INFO:teuthology.orchestra.run.smithi043.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T01:31:50.844 INFO:teuthology.orchestra.run.smithi043.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T01:31:50.844 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T01:31:50.844 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T01:31:50.845 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T01:31:50.845 INFO:teuthology.orchestra.run.smithi043.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T01:31:50.845 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T01:31:50.845 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T01:31:50.845 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T01:31:50.845 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T01:31:50.845 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T01:31:50.846 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T01:31:50.846 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T01:31:50.846 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T01:31:50.846 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T01:31:50.847 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T01:31:50.847 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T01:31:50.847 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T01:31:50.847 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T01:31:50.847 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T01:31:50.848 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:31:50.848 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T01:31:50.848 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T01:31:50.848 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T01:31:50.848 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T01:31:50.848 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T01:31:50.848 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T01:31:50.849 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T01:31:50.849 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T01:31:50.849 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T01:31:50.849 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T01:31:50.849 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T01:31:50.849 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T01:31:50.849 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T01:31:50.850 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T01:31:50.850 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T01:31:50.850 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T01:31:50.850 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T01:31:50.850 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T01:31:50.850 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:31:50.851 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T01:31:50.851 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T01:31:50.851 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T01:31:50.851 INFO:teuthology.orchestra.run.smithi043.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T01:31:50.851 INFO:teuthology.orchestra.run.smithi043.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T01:31:50.851 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T01:31:50.852 INFO:teuthology.orchestra.run.smithi043.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T01:31:50.852 INFO:teuthology.orchestra.run.smithi043.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T01:31:50.852 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T01:31:50.852 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T01:31:50.852 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T01:31:50.852 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T01:31:50.852 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T01:31:50.853 INFO:teuthology.orchestra.run.smithi043.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T01:31:50.853 INFO:teuthology.orchestra.run.smithi043.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T01:31:50.853 INFO:teuthology.orchestra.run.smithi043.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T01:31:50.853 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T01:31:50.853 INFO:teuthology.orchestra.run.smithi043.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T01:31:50.853 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling module streams: 2023-07-20T01:31:50.854 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql 10 2023-07-20T01:31:50.854 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:31:50.854 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2023-07-20T01:31:50.855 INFO:teuthology.orchestra.run.smithi043.stdout:=========================================================================================================================== 2023-07-20T01:31:50.855 INFO:teuthology.orchestra.run.smithi043.stdout:Install 128 Packages 2023-07-20T01:31:50.855 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 3 Packages 2023-07-20T01:31:50.855 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:31:50.856 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 206 M 2023-07-20T01:31:50.856 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2023-07-20T01:31:51.461 INFO:teuthology.orchestra.run.smithi043.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 29 kB/s | 6.4 kB 00:00 2023-07-20T01:31:53.237 INFO:teuthology.orchestra.run.smithi043.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 479 kB/s | 850 kB 00:01 2023-07-20T01:31:53.395 INFO:teuthology.orchestra.run.smithi043.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 1.0 MB/s | 167 kB 00:00 2023-07-20T01:31:54.837 INFO:teuthology.orchestra.run.smithi043.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 1.6 MB/s | 2.3 MB 00:01 2023-07-20T01:31:55.629 INFO:teuthology.orchestra.run.smithi043.stdout:(5/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 2.0 MB/s | 1.6 MB 00:00 2023-07-20T01:32:01.270 INFO:teuthology.orchestra.run.smithi043.stdout:(6/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 836 kB/s | 4.6 MB 00:05 2023-07-20T01:32:01.454 INFO:teuthology.orchestra.run.smithi043.stdout:(7/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 539 kB/s | 5.4 MB 00:10 2023-07-20T01:32:02.716 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:32:02.718 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:32:02.733 INFO:teuthology.orchestra.run.smithi003.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T01:32:02.733 INFO:teuthology.orchestra.run.smithi003.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T01:32:02.735 INFO:teuthology.orchestra.run.smithi003.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T01:32:02.736 INFO:teuthology.orchestra.run.smithi003.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T01:32:02.738 INFO:teuthology.orchestra.run.smithi003.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T01:32:02.742 INFO:teuthology.orchestra.run.smithi003.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T01:32:02.743 INFO:teuthology.orchestra.run.smithi003.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T01:32:02.890 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-07-20T01:32:02.903 INFO:teuthology.orchestra.run.smithi003.stdout:=========================================================================================================================== 2023-07-20T01:32:02.903 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-07-20T01:32:02.904 INFO:teuthology.orchestra.run.smithi003.stdout:=========================================================================================================================== 2023-07-20T01:32:02.904 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-07-20T01:32:02.904 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T01:32:02.904 INFO:teuthology.orchestra.run.smithi003.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T01:32:02.904 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T01:32:02.904 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T01:32:02.905 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T01:32:02.905 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T01:32:02.905 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T01:32:02.905 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T01:32:02.905 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T01:32:02.905 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T01:32:02.905 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T01:32:02.905 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T01:32:02.906 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T01:32:02.906 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T01:32:02.906 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T01:32:02.906 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T01:32:02.906 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T01:32:02.906 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T01:32:02.906 INFO:teuthology.orchestra.run.smithi003.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T01:32:02.907 INFO:teuthology.orchestra.run.smithi003.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T01:32:02.907 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T01:32:02.907 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T01:32:02.907 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:32:02.907 INFO:teuthology.orchestra.run.smithi003.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T01:32:02.907 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T01:32:02.907 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T01:32:02.907 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T01:32:02.908 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T01:32:02.908 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T01:32:02.908 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T01:32:02.908 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T01:32:02.908 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T01:32:02.908 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T01:32:02.908 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T01:32:02.909 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T01:32:02.909 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T01:32:02.909 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T01:32:02.909 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2023-07-20T01:32:02.909 INFO:teuthology.orchestra.run.smithi003.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T01:32:02.910 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T01:32:02.910 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T01:32:02.910 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-07-20T01:32:02.910 INFO:teuthology.orchestra.run.smithi003.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T01:32:02.910 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T01:32:02.910 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T01:32:02.910 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T01:32:02.910 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T01:32:02.911 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T01:32:02.911 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T01:32:02.911 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T01:32:02.912 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T01:32:02.912 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T01:32:02.912 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T01:32:02.912 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T01:32:02.912 INFO:teuthology.orchestra.run.smithi003.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T01:32:02.912 INFO:teuthology.orchestra.run.smithi003.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T01:32:02.912 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:32:02.913 INFO:teuthology.orchestra.run.smithi003.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T01:32:02.913 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T01:32:02.913 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T01:32:02.913 INFO:teuthology.orchestra.run.smithi003.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T01:32:02.913 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T01:32:02.913 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T01:32:02.913 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T01:32:02.914 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T01:32:02.914 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T01:32:02.914 INFO:teuthology.orchestra.run.smithi003.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T01:32:02.914 INFO:teuthology.orchestra.run.smithi003.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T01:32:02.914 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T01:32:02.914 INFO:teuthology.orchestra.run.smithi003.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T01:32:02.914 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T01:32:02.915 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T01:32:02.915 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T01:32:02.915 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T01:32:02.915 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T01:32:02.915 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T01:32:02.915 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T01:32:02.915 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T01:32:02.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T01:32:02.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T01:32:02.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T01:32:02.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T01:32:02.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T01:32:02.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T01:32:02.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T01:32:02.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T01:32:02.917 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T01:32:02.917 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T01:32:02.917 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T01:32:02.917 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T01:32:02.917 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T01:32:02.917 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T01:32:02.917 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T01:32:02.917 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T01:32:02.918 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T01:32:02.918 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:32:02.918 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T01:32:02.918 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T01:32:02.918 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T01:32:02.918 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T01:32:02.918 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T01:32:02.919 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T01:32:02.919 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T01:32:02.919 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T01:32:02.919 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T01:32:02.919 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T01:32:02.919 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T01:32:02.919 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T01:32:02.919 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T01:32:02.920 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T01:32:02.920 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T01:32:02.920 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T01:32:02.920 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T01:32:02.920 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T01:32:02.920 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:32:02.920 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T01:32:02.920 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T01:32:02.921 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T01:32:02.921 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T01:32:02.921 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T01:32:02.921 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T01:32:02.922 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T01:32:02.922 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T01:32:02.922 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T01:32:02.922 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T01:32:02.922 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T01:32:02.923 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T01:32:02.923 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T01:32:02.923 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T01:32:02.923 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T01:32:02.923 INFO:teuthology.orchestra.run.smithi003.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T01:32:02.923 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T01:32:02.923 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T01:32:02.924 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling module streams: 2023-07-20T01:32:02.924 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql 10 2023-07-20T01:32:02.924 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:32:02.924 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-07-20T01:32:02.924 INFO:teuthology.orchestra.run.smithi003.stdout:=========================================================================================================================== 2023-07-20T01:32:02.924 INFO:teuthology.orchestra.run.smithi003.stdout:Install 128 Packages 2023-07-20T01:32:02.924 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 3 Packages 2023-07-20T01:32:02.924 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:32:02.925 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 206 M 2023-07-20T01:32:02.925 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-07-20T01:32:03.802 INFO:teuthology.orchestra.run.smithi003.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 17 kB/s | 6.4 kB 00:00 2023-07-20T01:32:05.393 INFO:teuthology.orchestra.run.smithi003.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 534 kB/s | 850 kB 00:01 2023-07-20T01:32:05.585 INFO:teuthology.orchestra.run.smithi003.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 872 kB/s | 167 kB 00:00 2023-07-20T01:32:06.960 INFO:teuthology.orchestra.run.smithi003.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 1.7 MB/s | 2.3 MB 00:01 2023-07-20T01:32:07.297 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 370 kB/s | 16 MB 00:43 2023-07-20T01:32:07.952 INFO:teuthology.orchestra.run.smithi003.stdout:(5/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 1.2 MB/s | 5.4 MB 00:04 2023-07-20T01:32:10.119 INFO:teuthology.orchestra.run.smithi003.stdout:(6/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 503 kB/s | 1.6 MB 00:03 2023-07-20T01:32:10.728 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 429 kB/s | 24 kB 00:00 2023-07-20T01:32:11.804 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 53 MB/s | 53 MB 00:01 2023-07-20T01:32:13.252 INFO:teuthology.orchestra.run.smithi003.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 890 kB/s | 4.6 MB 00:05 2023-07-20T01:32:14.211 INFO:teuthology.orchestra.run.smithi043.stdout:(8/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el8 1.1 MB/s | 14 MB 00:12 2023-07-20T01:32:14.295 INFO:teuthology.orchestra.run.smithi043.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 292 kB/s | 24 kB 00:00 2023-07-20T01:32:16.270 INFO:teuthology.orchestra.run.smithi043.stdout:(10/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8 907 kB/s | 22 MB 00:25 2023-07-20T01:32:16.662 INFO:teuthology.orchestra.run.smithi043.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 665 kB/s | 260 kB 00:00 2023-07-20T01:32:16.762 INFO:teuthology.orchestra.run.smithi043.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 311 kB/s | 31 kB 00:00 2023-07-20T01:32:17.328 INFO:teuthology.orchestra.run.smithi043.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 1.3 MB/s | 771 kB 00:00 2023-07-20T01:32:17.512 INFO:teuthology.orchestra.run.smithi043.stdout:(14/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 1.1 MB/s | 18 MB 00:16 2023-07-20T01:32:17.537 INFO:teuthology.orchestra.run.smithi043.stdout:(15/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 666 kB/s | 139 kB 00:00 2023-07-20T01:32:17.654 INFO:teuthology.orchestra.run.smithi043.stdout:(16/131): librados-devel-18.0.0-5040.g1b0b2eb0. 982 kB/s | 139 kB 00:00 2023-07-20T01:32:18.155 INFO:teuthology.orchestra.run.smithi043.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 765 kB/s | 472 kB 00:00 2023-07-20T01:32:18.247 INFO:teuthology.orchestra.run.smithi043.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 489 kB/s | 45 kB 00:00 2023-07-20T01:32:18.439 INFO:teuthology.orchestra.run.smithi043.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 640 kB/s | 122 kB 00:00 2023-07-20T01:32:18.569 INFO:teuthology.orchestra.run.smithi003.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 1.5 MB/s | 22 MB 00:15 2023-07-20T01:32:18.660 INFO:teuthology.orchestra.run.smithi003.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 266 kB/s | 24 kB 00:00 2023-07-20T01:32:18.664 INFO:teuthology.orchestra.run.smithi043.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 875 kB/s | 197 kB 00:00 2023-07-20T01:32:18.972 INFO:teuthology.orchestra.run.smithi043.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 1.2 MB/s | 370 kB 00:00 2023-07-20T01:32:19.156 INFO:teuthology.orchestra.run.smithi043.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 1.9 MB/s | 363 kB 00:00 2023-07-20T01:32:19.256 INFO:teuthology.orchestra.run.smithi043.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.0 MB/s | 107 kB 00:00 2023-07-20T01:32:19.323 INFO:teuthology.orchestra.run.smithi043.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.4 MB/s | 94 kB 00:00 2023-07-20T01:32:19.847 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 MB/s | 47 MB 00:00 2023-07-20T01:32:21.127 INFO:teuthology.orchestra.run.smithi003.stdout:(10/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 1.6 MB/s | 18 MB 00:11 2023-07-20T01:32:21.148 INFO:teuthology.orchestra.run.smithi043.stdout:(25/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 1.9 MB/s | 3.5 MB 00:01 2023-07-20T01:32:21.265 INFO:teuthology.orchestra.run.smithi043.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.5 MB/s | 179 kB 00:00 2023-07-20T01:32:21.277 INFO:teuthology.orchestra.run.smithi003.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 1.7 MB/s | 260 kB 00:00 2023-07-20T01:32:21.336 INFO:teuthology.orchestra.run.smithi003.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 534 kB/s | 31 kB 00:00 2023-07-20T01:32:21.340 INFO:teuthology.orchestra.run.smithi043.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 306 kB/s | 23 kB 00:00 2023-07-20T01:32:21.441 INFO:teuthology.orchestra.run.smithi043.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.3 MB/s | 129 kB 00:00 2023-07-20T01:32:21.487 INFO:teuthology.orchestra.run.smithi003.stdout:(13/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 1.6 MB/s | 14 MB 00:08 2023-07-20T01:32:21.611 INFO:teuthology.orchestra.run.smithi003.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.1 MB/s | 139 kB 00:00 2023-07-20T01:32:21.720 INFO:teuthology.orchestra.run.smithi003.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.3 MB/s | 139 kB 00:00 2023-07-20T01:32:21.787 INFO:teuthology.orchestra.run.smithi003.stdout:(16/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 1.7 MB/s | 771 kB 00:00 2023-07-20T01:32:22.075 INFO:teuthology.orchestra.run.smithi043.stdout:(29/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 1.5 MB/s | 6.8 MB 00:04 2023-07-20T01:32:22.095 INFO:teuthology.orchestra.run.smithi003.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 1.2 MB/s | 472 kB 00:00 2023-07-20T01:32:22.158 INFO:teuthology.orchestra.run.smithi043.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 2.4 MB/s | 1.7 MB 00:00 2023-07-20T01:32:22.171 INFO:teuthology.orchestra.run.smithi003.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 597 kB/s | 45 kB 00:00 2023-07-20T01:32:22.283 INFO:teuthology.orchestra.run.smithi043.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 1.9 MB/s | 241 kB 00:00 2023-07-20T01:32:22.287 INFO:teuthology.orchestra.run.smithi003.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 1.0 MB/s | 122 kB 00:00 2023-07-20T01:32:22.359 INFO:teuthology.orchestra.run.smithi043.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 634 kB/s | 47 kB 00:00 2023-07-20T01:32:22.434 INFO:teuthology.orchestra.run.smithi043.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 194 kB/s | 15 kB 00:00 2023-07-20T01:32:22.584 INFO:teuthology.orchestra.run.smithi043.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 1.7 MB/s | 258 kB 00:00 2023-07-20T01:32:22.630 INFO:teuthology.orchestra.run.smithi003.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 576 kB/s | 197 kB 00:00 2023-07-20T01:32:22.710 INFO:teuthology.orchestra.run.smithi043.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 1.3 MB/s | 165 kB 00:00 2023-07-20T01:32:22.793 INFO:teuthology.orchestra.run.smithi043.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 216 kB/s | 18 kB 00:00 2023-07-20T01:32:22.996 INFO:teuthology.orchestra.run.smithi003.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 1.0 MB/s | 370 kB 00:00 2023-07-20T01:32:23.747 INFO:teuthology.orchestra.run.smithi003.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 484 kB/s | 363 kB 00:00 2023-07-20T01:32:23.989 INFO:teuthology.orchestra.run.smithi003.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 441 kB/s | 107 kB 00:00 2023-07-20T01:32:24.180 INFO:teuthology.orchestra.run.smithi003.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 490 kB/s | 94 kB 00:00 2023-07-20T01:32:24.660 INFO:teuthology.orchestra.run.smithi043.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 164 kB/s | 306 kB 00:01 2023-07-20T01:32:24.860 INFO:teuthology.orchestra.run.smithi043.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 294 kB/s | 59 kB 00:00 2023-07-20T01:32:25.060 INFO:teuthology.orchestra.run.smithi043.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 376 kB/s | 75 kB 00:00 2023-07-20T01:32:25.756 INFO:teuthology.orchestra.run.smithi003.stdout:(25/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 1.7 MB/s | 6.8 MB 00:03 2023-07-20T01:32:25.810 INFO:teuthology.orchestra.run.smithi043.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 634 kB/s | 476 kB 00:00 2023-07-20T01:32:25.864 INFO:teuthology.orchestra.run.smithi003.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.6 MB/s | 179 kB 00:00 2023-07-20T01:32:25.919 INFO:teuthology.orchestra.run.smithi043.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 403 kB/s | 44 kB 00:00 2023-07-20T01:32:25.931 INFO:teuthology.orchestra.run.smithi003.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 346 kB/s | 23 kB 00:00 2023-07-20T01:32:26.084 INFO:teuthology.orchestra.run.smithi003.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 847 kB/s | 129 kB 00:00 2023-07-20T01:32:26.245 INFO:teuthology.orchestra.run.smithi043.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 570 kB/s | 185 kB 00:00 2023-07-20T01:32:26.511 INFO:teuthology.orchestra.run.smithi043.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 124 kB/s | 33 kB 00:00 2023-07-20T01:32:26.595 INFO:teuthology.orchestra.run.smithi043.stdout:(44/131): ceph-mgr-diskprediction-local-18.0.0- 1.6 MB/s | 7.4 MB 00:04 2023-07-20T01:32:26.621 INFO:teuthology.orchestra.run.smithi043.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 150 kB/s | 16 kB 00:00 2023-07-20T01:32:26.948 INFO:teuthology.orchestra.run.smithi003.stdout:(29/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 2.0 MB/s | 1.7 MB 00:00 2023-07-20T01:32:26.962 INFO:teuthology.orchestra.run.smithi043.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 471 kB/s | 173 kB 00:00 2023-07-20T01:32:27.087 INFO:teuthology.orchestra.run.smithi043.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 650 kB/s | 81 kB 00:00 2023-07-20T01:32:27.155 INFO:teuthology.orchestra.run.smithi043.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2023-07-20T01:32:27.182 INFO:teuthology.orchestra.run.smithi003.stdout:(30/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 1.2 MB/s | 3.5 MB 00:03 2023-07-20T01:32:27.238 INFO:teuthology.orchestra.run.smithi043.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 211 kB/s | 18 kB 00:00 2023-07-20T01:32:27.416 INFO:teuthology.orchestra.run.smithi003.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 1.0 MB/s | 241 kB 00:00 2023-07-20T01:32:27.516 INFO:teuthology.orchestra.run.smithi003.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 474 kB/s | 47 kB 00:00 2023-07-20T01:32:27.616 INFO:teuthology.orchestra.run.smithi003.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 145 kB/s | 15 kB 00:00 2023-07-20T01:32:27.630 INFO:teuthology.orchestra.run.smithi043.stdout:(50/131): python3-cherrypy-18.4.0-1.el8.noarch. 381 kB/s | 384 kB 00:01 2023-07-20T01:32:27.789 INFO:teuthology.orchestra.run.smithi043.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 308 kB/s | 49 kB 00:00 2023-07-20T01:32:27.816 INFO:teuthology.orchestra.run.smithi003.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 1.3 MB/s | 258 kB 00:00 2023-07-20T01:32:27.947 INFO:teuthology.orchestra.run.smithi043.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 371 kB/s | 59 kB 00:00 2023-07-20T01:32:28.008 INFO:teuthology.orchestra.run.smithi003.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 862 kB/s | 165 kB 00:00 2023-07-20T01:32:28.075 INFO:teuthology.orchestra.run.smithi003.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 269 kB/s | 18 kB 00:00 2023-07-20T01:32:28.097 INFO:teuthology.orchestra.run.smithi043.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 402 kB/s | 60 kB 00:00 2023-07-20T01:32:28.390 INFO:teuthology.orchestra.run.smithi043.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 1.0 MB/s | 1.2 MB 00:01 2023-07-20T01:32:28.457 INFO:teuthology.orchestra.run.smithi043.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 234 kB/s | 16 kB 00:00 2023-07-20T01:32:28.532 INFO:teuthology.orchestra.run.smithi043.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 448 kB/s | 34 kB 00:00 2023-07-20T01:32:28.716 INFO:teuthology.orchestra.run.smithi043.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 458 kB/s | 283 kB 00:00 2023-07-20T01:32:28.783 INFO:teuthology.orchestra.run.smithi043.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 785 kB/s | 196 kB 00:00 2023-07-20T01:32:28.841 INFO:teuthology.orchestra.run.smithi043.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 491 kB/s | 61 kB 00:00 2023-07-20T01:32:28.866 INFO:teuthology.orchestra.run.smithi043.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 224 kB/s | 19 kB 00:00 2023-07-20T01:32:28.925 INFO:teuthology.orchestra.run.smithi043.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 286 kB/s | 24 kB 00:00 2023-07-20T01:32:28.967 INFO:teuthology.orchestra.run.smithi043.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 292 kB/s | 29 kB 00:00 2023-07-20T01:32:29.009 INFO:teuthology.orchestra.run.smithi043.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 330 kB/s | 27 kB 00:00 2023-07-20T01:32:29.060 INFO:teuthology.orchestra.run.smithi003.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 311 kB/s | 306 kB 00:00 2023-07-20T01:32:29.193 INFO:teuthology.orchestra.run.smithi003.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 442 kB/s | 59 kB 00:00 2023-07-20T01:32:29.251 INFO:teuthology.orchestra.run.smithi043.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 901 kB/s | 255 kB 00:00 2023-07-20T01:32:29.309 INFO:teuthology.orchestra.run.smithi003.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 645 kB/s | 75 kB 00:00 2023-07-20T01:32:29.318 INFO:teuthology.orchestra.run.smithi043.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 919 kB/s | 61 kB 00:00 2023-07-20T01:32:29.468 INFO:teuthology.orchestra.run.smithi043.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 547 kB/s | 251 kB 00:00 2023-07-20T01:32:29.494 INFO:teuthology.orchestra.run.smithi043.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 487 kB/s | 85 kB 00:00 2023-07-20T01:32:29.535 INFO:teuthology.orchestra.run.smithi043.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 344 kB/s | 23 kB 00:00 2023-07-20T01:32:29.561 INFO:teuthology.orchestra.run.smithi043.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T01:32:29.628 INFO:teuthology.orchestra.run.smithi043.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 704 kB/s | 47 kB 00:00 2023-07-20T01:32:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-07-20T01:32:29.712 INFO:teuthology.orchestra.run.smithi043.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-07-20T01:32:29.754 INFO:teuthology.orchestra.run.smithi043.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-07-20T01:32:29.796 INFO:teuthology.orchestra.run.smithi043.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.1 MB/s | 48 kB 00:00 2023-07-20T01:32:29.854 INFO:teuthology.orchestra.run.smithi043.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.5 MB/s | 87 kB 00:00 2023-07-20T01:32:29.896 INFO:teuthology.orchestra.run.smithi043.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.0 MB/s | 43 kB 00:00 2023-07-20T01:32:29.923 INFO:teuthology.orchestra.run.smithi003.stdout:(40/131): ceph-mgr-diskprediction-local-18.0.0- 2.5 MB/s | 7.4 MB 00:02 2023-07-20T01:32:29.938 INFO:teuthology.orchestra.run.smithi043.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 2.6 MB/s | 109 kB 00:00 2023-07-20T01:32:29.980 INFO:teuthology.orchestra.run.smithi043.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2023-07-20T01:32:30.026 INFO:teuthology.orchestra.run.smithi003.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 664 kB/s | 476 kB 00:00 2023-07-20T01:32:30.030 INFO:teuthology.orchestra.run.smithi043.stdout:(79/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 11 MB/s | 560 kB 00:00 2023-07-20T01:32:30.081 INFO:teuthology.orchestra.run.smithi043.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 372 kB/s | 19 kB 00:00 2023-07-20T01:32:30.115 INFO:teuthology.orchestra.run.smithi043.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 4.6 MB/s | 155 kB 00:00 2023-07-20T01:32:30.165 INFO:teuthology.orchestra.run.smithi043.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 11 MB/s | 559 kB 00:00 2023-07-20T01:32:30.215 INFO:teuthology.orchestra.run.smithi043.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 358 kB/s | 18 kB 00:00 2023-07-20T01:32:30.266 INFO:teuthology.orchestra.run.smithi043.stdout:(84/131): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2023-07-20T01:32:30.316 INFO:teuthology.orchestra.run.smithi043.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.6 MB/s | 238 kB 00:00 2023-07-20T01:32:30.327 INFO:teuthology.orchestra.run.smithi003.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 627 kB/s | 185 kB 00:00 2023-07-20T01:32:30.408 INFO:teuthology.orchestra.run.smithi043.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 25 MB/s | 2.3 MB 00:00 2023-07-20T01:32:30.442 INFO:teuthology.orchestra.run.smithi043.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 2.1 MB/s | 72 kB 00:00 2023-07-20T01:32:30.452 INFO:teuthology.orchestra.run.smithi003.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 264 kB/s | 33 kB 00:00 2023-07-20T01:32:30.484 INFO:teuthology.orchestra.run.smithi043.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2023-07-20T01:32:30.525 INFO:teuthology.orchestra.run.smithi043.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 1.4 MB/s | 60 kB 00:00 2023-07-20T01:32:30.566 INFO:teuthology.orchestra.run.smithi003.stdout:(44/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 68 kB/s | 44 kB 00:00 2023-07-20T01:32:30.567 INFO:teuthology.orchestra.run.smithi043.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 2.3 MB/s | 98 kB 00:00 2023-07-20T01:32:30.595 INFO:teuthology.orchestra.run.smithi003.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 115 kB/s | 16 kB 00:00 2023-07-20T01:32:30.609 INFO:teuthology.orchestra.run.smithi043.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2023-07-20T01:32:30.652 INFO:teuthology.orchestra.run.smithi043.stdout:(92/131): libcom_err-devel-1.45.6-4.el8.x86_64. 924 kB/s | 39 kB 00:00 2023-07-20T01:32:30.710 INFO:teuthology.orchestra.run.smithi043.stdout:(93/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 11 MB/s | 644 kB 00:00 2023-07-20T01:32:30.736 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:32:30.737 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:32:30.751 INFO:teuthology.orchestra.run.smithi086.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T01:32:30.752 INFO:teuthology.orchestra.run.smithi043.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 4.0 MB/s | 171 kB 00:00 2023-07-20T01:32:30.753 INFO:teuthology.orchestra.run.smithi086.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T01:32:30.753 INFO:teuthology.orchestra.run.smithi086.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T01:32:30.755 INFO:teuthology.orchestra.run.smithi086.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T01:32:30.757 INFO:teuthology.orchestra.run.smithi086.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T01:32:30.760 INFO:teuthology.orchestra.run.smithi086.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T01:32:30.761 INFO:teuthology.orchestra.run.smithi086.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T01:32:30.794 INFO:teuthology.orchestra.run.smithi043.stdout:(95/131): NetworkManager-initscripts-updown-1.3 3.2 MB/s | 138 kB 00:00 2023-07-20T01:32:30.845 INFO:teuthology.orchestra.run.smithi043.stdout:(96/131): thrift-0.13.0-2.el8.x86_64.rpm 1.3 MB/s | 1.7 MB 00:01 2023-07-20T01:32:30.878 INFO:teuthology.orchestra.run.smithi003.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 554 kB/s | 173 kB 00:00 2023-07-20T01:32:30.895 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-07-20T01:32:30.907 INFO:teuthology.orchestra.run.smithi086.stdout:=========================================================================================================================== 2023-07-20T01:32:30.907 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-07-20T01:32:30.907 INFO:teuthology.orchestra.run.smithi086.stdout:=========================================================================================================================== 2023-07-20T01:32:30.907 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-07-20T01:32:30.908 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T01:32:30.908 INFO:teuthology.orchestra.run.smithi086.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T01:32:30.908 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T01:32:30.908 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T01:32:30.908 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T01:32:30.909 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T01:32:30.909 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T01:32:30.909 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T01:32:30.909 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T01:32:30.909 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T01:32:30.909 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T01:32:30.909 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T01:32:30.910 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T01:32:30.910 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T01:32:30.910 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T01:32:30.910 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T01:32:30.910 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T01:32:30.910 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T01:32:30.910 INFO:teuthology.orchestra.run.smithi086.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T01:32:30.911 INFO:teuthology.orchestra.run.smithi086.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T01:32:30.911 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T01:32:30.911 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T01:32:30.911 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:32:30.912 INFO:teuthology.orchestra.run.smithi086.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T01:32:30.912 INFO:teuthology.orchestra.run.smithi086.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T01:32:30.912 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T01:32:30.912 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T01:32:30.912 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T01:32:30.913 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T01:32:30.913 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T01:32:30.913 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T01:32:30.913 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T01:32:30.914 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T01:32:30.914 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T01:32:30.914 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T01:32:30.914 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T01:32:30.914 INFO:teuthology.orchestra.run.smithi086.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T01:32:30.915 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2023-07-20T01:32:30.916 INFO:teuthology.orchestra.run.smithi086.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T01:32:30.916 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T01:32:30.916 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T01:32:30.916 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-07-20T01:32:30.916 INFO:teuthology.orchestra.run.smithi086.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T01:32:30.916 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T01:32:30.917 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T01:32:30.917 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T01:32:30.917 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T01:32:30.917 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T01:32:30.917 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T01:32:30.917 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T01:32:30.917 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T01:32:30.917 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T01:32:30.918 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T01:32:30.918 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T01:32:30.918 INFO:teuthology.orchestra.run.smithi086.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T01:32:30.918 INFO:teuthology.orchestra.run.smithi086.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T01:32:30.918 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:32:30.918 INFO:teuthology.orchestra.run.smithi086.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T01:32:30.918 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T01:32:30.919 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T01:32:30.919 INFO:teuthology.orchestra.run.smithi086.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T01:32:30.919 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T01:32:30.919 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T01:32:30.919 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T01:32:30.919 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T01:32:30.919 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T01:32:30.919 INFO:teuthology.orchestra.run.smithi086.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T01:32:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T01:32:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T01:32:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T01:32:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T01:32:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T01:32:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T01:32:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T01:32:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T01:32:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T01:32:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T01:32:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T01:32:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T01:32:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T01:32:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T01:32:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T01:32:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T01:32:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T01:32:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T01:32:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T01:32:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T01:32:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T01:32:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T01:32:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T01:32:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T01:32:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T01:32:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T01:32:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T01:32:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T01:32:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:32:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T01:32:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T01:32:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T01:32:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T01:32:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T01:32:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T01:32:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T01:32:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T01:32:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T01:32:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T01:32:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T01:32:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T01:32:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T01:32:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T01:32:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T01:32:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T01:32:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T01:32:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T01:32:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:32:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T01:32:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T01:32:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T01:32:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T01:32:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T01:32:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T01:32:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T01:32:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T01:32:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T01:32:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T01:32:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T01:32:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T01:32:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T01:32:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T01:32:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T01:32:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T01:32:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T01:32:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T01:32:30.928 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling module streams: 2023-07-20T01:32:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql 10 2023-07-20T01:32:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:32:30.929 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-07-20T01:32:30.929 INFO:teuthology.orchestra.run.smithi086.stdout:=========================================================================================================================== 2023-07-20T01:32:30.929 INFO:teuthology.orchestra.run.smithi086.stdout:Install 128 Packages 2023-07-20T01:32:30.929 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 3 Packages 2023-07-20T01:32:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:32:30.930 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 206 M 2023-07-20T01:32:30.930 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-07-20T01:32:30.937 INFO:teuthology.orchestra.run.smithi043.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.2 MB/s | 110 kB 00:00 2023-07-20T01:32:30.987 INFO:teuthology.orchestra.run.smithi043.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-07-20T01:32:31.037 INFO:teuthology.orchestra.run.smithi043.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-07-20T01:32:31.086 INFO:teuthology.orchestra.run.smithi003.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 390 kB/s | 81 kB 00:00 2023-07-20T01:32:31.088 INFO:teuthology.orchestra.run.smithi043.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-07-20T01:32:31.129 INFO:teuthology.orchestra.run.smithi043.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 935 kB/s | 39 kB 00:00 2023-07-20T01:32:31.180 INFO:teuthology.orchestra.run.smithi043.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-07-20T01:32:31.210 INFO:teuthology.orchestra.run.smithi003.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 625 kB/s | 384 kB 00:00 2023-07-20T01:32:31.229 INFO:teuthology.orchestra.run.smithi003.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2023-07-20T01:32:31.313 INFO:teuthology.orchestra.run.smithi086.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 33 kB/s | 6.4 kB 00:00 2023-07-20T01:32:31.321 INFO:teuthology.orchestra.run.smithi003.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 159 kB/s | 18 kB 00:00 2023-07-20T01:32:31.322 INFO:teuthology.orchestra.run.smithi043.stdout:(103/131): python3-scipy-1.0.0-21.module+el8.5. 26 MB/s | 14 MB 00:00 2023-07-20T01:32:31.355 INFO:teuthology.orchestra.run.smithi043.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 3.0 MB/s | 538 kB 00:00 2023-07-20T01:32:31.380 INFO:teuthology.orchestra.run.smithi043.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-07-20T01:32:31.398 INFO:teuthology.orchestra.run.smithi043.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-07-20T01:32:31.429 INFO:teuthology.orchestra.run.smithi003.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 450 kB/s | 49 kB 00:00 2023-07-20T01:32:31.431 INFO:teuthology.orchestra.run.smithi043.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 14 MB/s | 713 kB 00:00 2023-07-20T01:32:31.473 INFO:teuthology.orchestra.run.smithi043.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 636 kB/s | 27 kB 00:00 2023-07-20T01:32:31.516 INFO:teuthology.orchestra.run.smithi043.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-07-20T01:32:31.546 INFO:teuthology.orchestra.run.smithi003.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 504 kB/s | 59 kB 00:00 2023-07-20T01:32:31.591 INFO:teuthology.orchestra.run.smithi043.stdout:(110/131): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2023-07-20T01:32:31.616 INFO:teuthology.orchestra.run.smithi043.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 440 kB/s | 44 kB 00:00 2023-07-20T01:32:31.666 INFO:teuthology.orchestra.run.smithi043.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 8.9 MB/s | 457 kB 00:00 2023-07-20T01:32:31.672 INFO:teuthology.orchestra.run.smithi003.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 482 kB/s | 60 kB 00:00 2023-07-20T01:32:31.741 INFO:teuthology.orchestra.run.smithi043.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 25 MB/s | 3.7 MB 00:00 2023-07-20T01:32:31.767 INFO:teuthology.orchestra.run.smithi043.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 2.2 MB/s | 225 kB 00:00 2023-07-20T01:32:31.792 INFO:teuthology.orchestra.run.smithi043.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-07-20T01:32:31.841 INFO:teuthology.orchestra.run.smithi043.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.7 MB/s | 126 kB 00:00 2023-07-20T01:32:31.867 INFO:teuthology.orchestra.run.smithi043.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 265 kB/s | 20 kB 00:00 2023-07-20T01:32:31.884 INFO:teuthology.orchestra.run.smithi043.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.4 MB/s | 103 kB 00:00 2023-07-20T01:32:31.918 INFO:teuthology.orchestra.run.smithi043.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 14 MB/s | 709 kB 00:00 2023-07-20T01:32:31.960 INFO:teuthology.orchestra.run.smithi043.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.7 MB/s | 198 kB 00:00 2023-07-20T01:32:32.060 INFO:teuthology.orchestra.run.smithi043.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 28 MB/s | 4.9 MB 00:00 2023-07-20T01:32:32.072 INFO:teuthology.orchestra.run.smithi086.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.1 MB/s | 850 kB 00:00 2023-07-20T01:32:32.103 INFO:teuthology.orchestra.run.smithi043.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-07-20T01:32:32.177 INFO:teuthology.orchestra.run.smithi043.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2023-07-20T01:32:32.211 INFO:teuthology.orchestra.run.smithi043.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.2 MB/s | 354 kB 00:00 2023-07-20T01:32:32.231 INFO:teuthology.orchestra.run.smithi086.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 1.0 MB/s | 167 kB 00:00 2023-07-20T01:32:32.270 INFO:teuthology.orchestra.run.smithi043.stdout:(125/131): postgresql-contrib-10.21-2.module+el 14 MB/s | 808 kB 00:00 2023-07-20T01:32:32.288 INFO:teuthology.orchestra.run.smithi003.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 458 kB/s | 283 kB 00:00 2023-07-20T01:32:32.362 INFO:teuthology.orchestra.run.smithi043.stdout:(126/131): postgresql-server-10.21-2.module+el8 28 MB/s | 5.1 MB 00:00 2023-07-20T01:32:32.404 INFO:teuthology.orchestra.run.smithi043.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 11 MB/s | 1.5 MB 00:00 2023-07-20T01:32:32.430 INFO:teuthology.orchestra.run.smithi003.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 110 kB/s | 16 kB 00:00 2023-07-20T01:32:33.189 INFO:teuthology.orchestra.run.smithi003.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 44 kB/s | 34 kB 00:00 2023-07-20T01:32:33.564 INFO:teuthology.orchestra.run.smithi003.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 524 kB/s | 196 kB 00:00 2023-07-20T01:32:33.681 INFO:teuthology.orchestra.run.smithi003.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 528 kB/s | 61 kB 00:00 2023-07-20T01:32:33.798 INFO:teuthology.orchestra.run.smithi003.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 160 kB/s | 19 kB 00:00 2023-07-20T01:32:33.915 INFO:teuthology.orchestra.run.smithi003.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 205 kB/s | 24 kB 00:00 2023-07-20T01:32:33.947 INFO:teuthology.orchestra.run.smithi086.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 1.3 MB/s | 2.3 MB 00:01 2023-07-20T01:32:33.949 INFO:teuthology.orchestra.run.smithi003.stdout:(61/131): python3-kubernetes-11.0.0-6.el8.noarc 439 kB/s | 1.2 MB 00:02 2023-07-20T01:32:34.024 INFO:teuthology.orchestra.run.smithi003.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 269 kB/s | 29 kB 00:00 2023-07-20T01:32:34.058 INFO:teuthology.orchestra.run.smithi003.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 253 kB/s | 27 kB 00:00 2023-07-20T01:32:34.312 INFO:teuthology.orchestra.run.smithi043.stdout:(128/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8. 2.4 MB/s | 48 MB 00:20 2023-07-20T01:32:34.370 INFO:teuthology.orchestra.run.smithi043.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 13 MB/s | 798 kB 00:00 2023-07-20T01:32:34.537 INFO:teuthology.orchestra.run.smithi043.stdout:(130/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 1.7 MB/s | 3.6 MB 00:02 2023-07-20T01:32:34.550 INFO:teuthology.orchestra.run.smithi003.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 486 kB/s | 255 kB 00:00 2023-07-20T01:32:34.575 INFO:teuthology.orchestra.run.smithi003.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 486 kB/s | 251 kB 00:00 2023-07-20T01:32:34.700 INFO:teuthology.orchestra.run.smithi003.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 408 kB/s | 61 kB 00:00 2023-07-20T01:32:34.851 INFO:teuthology.orchestra.run.smithi003.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 153 kB/s | 23 kB 00:00 2023-07-20T01:32:34.867 INFO:teuthology.orchestra.run.smithi003.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 292 kB/s | 85 kB 00:00 2023-07-20T01:32:34.871 INFO:teuthology.orchestra.run.smithi043.stdout:(131/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 1.5 MB/s | 3.7 MB 00:02 2023-07-20T01:32:34.876 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:32:34.876 INFO:teuthology.orchestra.run.smithi043.stdout:Total 4.7 MB/s | 206 MB 00:44 2023-07-20T01:32:34.885 INFO:teuthology.orchestra.run.smithi003.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.1 MB/s | 70 kB 00:00 2023-07-20T01:32:35.218 INFO:teuthology.orchestra.run.smithi003.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 41 MB/s | 14 MB 00:00 2023-07-20T01:32:35.260 INFO:teuthology.orchestra.run.smithi003.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2023-07-20T01:32:35.302 INFO:teuthology.orchestra.run.smithi003.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-07-20T01:32:35.344 INFO:teuthology.orchestra.run.smithi003.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-20T01:32:35.364 INFO:teuthology.orchestra.run.smithi086.stdout:(5/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 1.1 MB/s | 1.6 MB 00:01 2023-07-20T01:32:35.394 INFO:teuthology.orchestra.run.smithi003.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-07-20T01:32:35.453 INFO:teuthology.orchestra.run.smithi003.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 666 kB/s | 39 kB 00:00 2023-07-20T01:32:35.503 INFO:teuthology.orchestra.run.smithi003.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-07-20T01:32:35.554 INFO:teuthology.orchestra.run.smithi003.stdout:(77/131): python3-jinja2-2.10.1-3.el8.noarch.rp 11 MB/s | 538 kB 00:00 2023-07-20T01:32:35.595 INFO:teuthology.orchestra.run.smithi003.stdout:(78/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-07-20T01:32:35.637 INFO:teuthology.orchestra.run.smithi003.stdout:(79/131): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-07-20T01:32:35.696 INFO:teuthology.orchestra.run.smithi003.stdout:(80/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-07-20T01:32:35.713 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2023-07-20T01:32:35.830 INFO:teuthology.orchestra.run.smithi003.stdout:(81/131): python3-babel-2.5.1-7.el8.noarch.rpm 36 MB/s | 4.8 MB 00:00 2023-07-20T01:32:35.845 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2023-07-20T01:32:35.845 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2023-07-20T01:32:35.863 INFO:teuthology.orchestra.run.smithi003.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 801 kB/s | 27 kB 00:00 2023-07-20T01:32:35.913 INFO:teuthology.orchestra.run.smithi003.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2023-07-20T01:32:35.956 INFO:teuthology.orchestra.run.smithi003.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 1.0 MB/s | 44 kB 00:00 2023-07-20T01:32:36.064 INFO:teuthology.orchestra.run.smithi003.stdout:(85/131): python3-numpy-1.14.3-10.el8.x86_64.rp 34 MB/s | 3.7 MB 00:00 2023-07-20T01:32:36.115 INFO:teuthology.orchestra.run.smithi003.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 8.9 MB/s | 457 kB 00:00 2023-07-20T01:32:36.157 INFO:teuthology.orchestra.run.smithi003.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2023-07-20T01:32:36.199 INFO:teuthology.orchestra.run.smithi003.stdout:(88/131): uuid-1.6.2-43.el8.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2023-07-20T01:32:36.241 INFO:teuthology.orchestra.run.smithi003.stdout:(89/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.0 MB/s | 126 kB 00:00 2023-07-20T01:32:36.283 INFO:teuthology.orchestra.run.smithi003.stdout:(90/131): python3-jsonpointer-1.10-11.el8.noarc 482 kB/s | 20 kB 00:00 2023-07-20T01:32:36.341 INFO:teuthology.orchestra.run.smithi003.stdout:(91/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.7 MB/s | 103 kB 00:00 2023-07-20T01:32:36.400 INFO:teuthology.orchestra.run.smithi003.stdout:(92/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-07-20T01:32:36.533 INFO:teuthology.orchestra.run.smithi003.stdout:(93/131): openblas-threads-0.3.15-3.el8.x86_64. 36 MB/s | 4.9 MB 00:00 2023-07-20T01:32:36.575 INFO:teuthology.orchestra.run.smithi003.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 4.6 MB/s | 198 kB 00:00 2023-07-20T01:32:36.709 INFO:teuthology.orchestra.run.smithi003.stdout:(95/131): openblas-0.3.15-3.el8.x86_64.rpm 35 MB/s | 4.7 MB 00:00 2023-07-20T01:32:36.776 INFO:teuthology.orchestra.run.smithi003.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 3.0 MB/s | 202 kB 00:00 2023-07-20T01:32:36.826 INFO:teuthology.orchestra.run.smithi003.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-07-20T01:32:36.977 INFO:teuthology.orchestra.run.smithi003.stdout:(98/131): postgresql-server-10.21-2.module+el8. 34 MB/s | 5.1 MB 00:00 2023-07-20T01:32:36.998 INFO:teuthology.orchestra.run.smithi086.stdout:(6/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 937 kB/s | 5.4 MB 00:05 2023-07-20T01:32:37.023 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2023-07-20T01:32:37.027 INFO:teuthology.orchestra.run.smithi003.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 16 MB/s | 808 kB 00:00 2023-07-20T01:32:37.027 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2023-07-20T01:32:37.094 INFO:teuthology.orchestra.run.smithi003.stdout:(100/131): postgresql-10.21-2.module+el8.6.0+15 23 MB/s | 1.5 MB 00:00 2023-07-20T01:32:37.127 INFO:teuthology.orchestra.run.smithi003.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2023-07-20T01:32:37.169 INFO:teuthology.orchestra.run.smithi003.stdout:(102/131): libselinux-devel-2.9-5.el8.x86_64.rp 4.7 MB/s | 200 kB 00:00 2023-07-20T01:32:37.211 INFO:teuthology.orchestra.run.smithi003.stdout:(103/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-07-20T01:32:37.253 INFO:teuthology.orchestra.run.smithi003.stdout:(104/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-07-20T01:32:37.296 INFO:teuthology.orchestra.run.smithi003.stdout:(105/131): keyutils-libs-devel-1.5.10-9.el8.x86 1.1 MB/s | 48 kB 00:00 2023-07-20T01:32:37.337 INFO:teuthology.orchestra.run.smithi003.stdout:(106/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-07-20T01:32:37.379 INFO:teuthology.orchestra.run.smithi003.stdout:(107/131): python3-requests-oauthlib-1.0.0-1.el 1.0 MB/s | 43 kB 00:00 2023-07-20T01:32:37.429 INFO:teuthology.orchestra.run.smithi003.stdout:(108/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-07-20T01:32:37.471 INFO:teuthology.orchestra.run.smithi003.stdout:(109/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2023-07-20T01:32:37.530 INFO:teuthology.orchestra.run.smithi003.stdout:(110/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.4 MB/s | 560 kB 00:00 2023-07-20T01:32:37.610 INFO:teuthology.orchestra.run.smithi003.stdout:(111/131): libaio-devel-0.3.112-1.el8.x86_64.rp 236 kB/s | 19 kB 00:00 2023-07-20T01:32:37.656 INFO:teuthology.orchestra.run.smithi003.stdout:(112/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.3 MB/s | 155 kB 00:00 2023-07-20T01:32:37.715 INFO:teuthology.orchestra.run.smithi003.stdout:(113/131): python3-cryptography-3.2.1-5.el8.x86 9.3 MB/s | 559 kB 00:00 2023-07-20T01:32:37.757 INFO:teuthology.orchestra.run.smithi003.stdout:(114/131): libverto-devel-0.3.0-5.el8.x86_64.rp 431 kB/s | 18 kB 00:00 2023-07-20T01:32:37.781 INFO:teuthology.orchestra.run.smithi086.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 1.9 MB/s | 4.6 MB 00:02 2023-07-20T01:32:37.807 INFO:teuthology.orchestra.run.smithi003.stdout:(115/131): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2023-07-20T01:32:37.840 INFO:teuthology.orchestra.run.smithi003.stdout:(116/131): thrift-0.13.0-2.el8.x86_64.rpm 594 kB/s | 1.7 MB 00:02 2023-07-20T01:32:37.866 INFO:teuthology.orchestra.run.smithi003.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2023-07-20T01:32:37.916 INFO:teuthology.orchestra.run.smithi003.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.7 MB/s | 72 kB 00:00 2023-07-20T01:32:37.958 INFO:teuthology.orchestra.run.smithi003.stdout:(119/131): fuse-2.9.7-15.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2023-07-20T01:32:38.008 INFO:teuthology.orchestra.run.smithi003.stdout:(120/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 14 MB/s | 2.3 MB 00:00 2023-07-20T01:32:38.033 INFO:teuthology.orchestra.run.smithi003.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 796 kB/s | 60 kB 00:00 2023-07-20T01:32:38.059 INFO:teuthology.orchestra.run.smithi003.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-07-20T01:32:38.084 INFO:teuthology.orchestra.run.smithi003.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-07-20T01:32:38.109 INFO:teuthology.orchestra.run.smithi003.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 770 kB/s | 39 kB 00:00 2023-07-20T01:32:38.151 INFO:teuthology.orchestra.run.smithi003.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 9.4 MB/s | 644 kB 00:00 2023-07-20T01:32:38.168 INFO:teuthology.orchestra.run.smithi003.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.9 MB/s | 171 kB 00:00 2023-07-20T01:32:38.194 INFO:teuthology.orchestra.run.smithi003.stdout:(127/131): NetworkManager-initscripts-updown-1. 3.2 MB/s | 138 kB 00:00 2023-07-20T01:32:40.194 INFO:teuthology.orchestra.run.smithi003.stdout:(128/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 1.8 MB/s | 3.7 MB 00:02 2023-07-20T01:32:40.252 INFO:teuthology.orchestra.run.smithi003.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 14 MB/s | 798 kB 00:00 2023-07-20T01:32:41.102 INFO:teuthology.orchestra.run.smithi003.stdout:(130/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 1.2 MB/s | 3.6 MB 00:02 2023-07-20T01:32:42.056 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2023-07-20T01:32:42.227 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T01:32:42.404 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T01:32:42.562 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T01:32:42.577 INFO:teuthology.orchestra.run.smithi003.stdout:(131/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8. 2.0 MB/s | 48 MB 00:23 2023-07-20T01:32:42.581 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:32:42.582 INFO:teuthology.orchestra.run.smithi003.stdout:Total 5.2 MB/s | 206 MB 00:39 2023-07-20T01:32:42.664 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T01:32:42.870 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T01:32:43.298 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T01:32:43.405 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-07-20T01:32:43.470 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T01:32:43.537 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-07-20T01:32:43.537 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-07-20T01:32:43.888 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T01:32:44.061 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:32:44.191 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:32:44.239 INFO:teuthology.orchestra.run.smithi086.stdout:(8/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el8 2.1 MB/s | 14 MB 00:06 2023-07-20T01:32:44.323 INFO:teuthology.orchestra.run.smithi086.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 293 kB/s | 24 kB 00:00 2023-07-20T01:32:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:32:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:32:44.658 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:32:44.724 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-07-20T01:32:44.728 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-07-20T01:32:44.813 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T01:32:45.046 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T01:32:45.241 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T01:32:45.663 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T01:32:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T01:32:46.036 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:32:46.143 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:32:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:32:46.439 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:32:46.683 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T01:32:46.723 INFO:teuthology.orchestra.run.smithi086.stdout:(10/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8 1.4 MB/s | 22 MB 00:15 2023-07-20T01:32:46.858 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T01:32:47.045 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T01:32:47.073 INFO:teuthology.orchestra.run.smithi086.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 744 kB/s | 260 kB 00:00 2023-07-20T01:32:47.140 INFO:teuthology.orchestra.run.smithi086.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 466 kB/s | 31 kB 00:00 2023-07-20T01:32:47.169 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T01:32:47.317 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T01:32:47.469 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T01:32:48.113 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T01:32:48.115 INFO:teuthology.orchestra.run.smithi086.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 792 kB/s | 771 kB 00:00 2023-07-20T01:32:48.257 INFO:teuthology.orchestra.run.smithi086.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 980 kB/s | 139 kB 00:00 2023-07-20T01:32:48.269 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T01:32:48.399 INFO:teuthology.orchestra.run.smithi086.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 982 kB/s | 139 kB 00:00 2023-07-20T01:32:48.782 INFO:teuthology.orchestra.run.smithi086.stdout:(16/131): libradosstriper1-18.0.0-5040.g1b0b2eb 1.2 MB/s | 472 kB 00:00 2023-07-20T01:32:48.894 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T01:32:49.077 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T01:32:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-07-20T01:32:49.809 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T01:32:49.848 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T01:32:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T01:32:50.059 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T01:32:50.217 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-20T01:32:50.232 INFO:teuthology.orchestra.run.smithi086.stdout:(17/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 1.3 MB/s | 18 MB 00:13 2023-07-20T01:32:50.239 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T01:32:50.291 INFO:teuthology.orchestra.run.smithi086.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 771 kB/s | 45 kB 00:00 2023-07-20T01:32:50.358 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T01:32:50.391 INFO:teuthology.orchestra.run.smithi086.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 1.2 MB/s | 122 kB 00:00 2023-07-20T01:32:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T01:32:50.475 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T01:32:50.525 INFO:teuthology.orchestra.run.smithi086.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 1.4 MB/s | 197 kB 00:00 2023-07-20T01:32:50.634 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T01:32:50.708 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T01:32:50.742 INFO:teuthology.orchestra.run.smithi086.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 1.7 MB/s | 370 kB 00:00 2023-07-20T01:32:50.765 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T01:32:50.951 INFO:teuthology.orchestra.run.smithi086.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 1.7 MB/s | 363 kB 00:00 2023-07-20T01:32:50.982 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T01:32:51.068 INFO:teuthology.orchestra.run.smithi086.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 915 kB/s | 107 kB 00:00 2023-07-20T01:32:51.134 INFO:teuthology.orchestra.run.smithi086.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.4 MB/s | 94 kB 00:00 2023-07-20T01:32:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T01:32:51.202 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T01:32:51.458 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:32:51.878 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:32:52.057 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:32:52.173 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:32:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:32:52.467 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:32:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T01:32:52.726 INFO:teuthology.orchestra.run.smithi086.stdout:(25/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 1.7 MB/s | 6.8 MB 00:03 2023-07-20T01:32:52.843 INFO:teuthology.orchestra.run.smithi086.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.5 MB/s | 179 kB 00:00 2023-07-20T01:32:52.884 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T01:32:52.910 INFO:teuthology.orchestra.run.smithi086.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 346 kB/s | 23 kB 00:00 2023-07-20T01:32:53.069 INFO:teuthology.orchestra.run.smithi086.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 817 kB/s | 129 kB 00:00 2023-07-20T01:32:53.091 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T01:32:53.111 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T01:32:53.319 INFO:teuthology.orchestra.run.smithi086.stdout:(29/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 1.6 MB/s | 3.5 MB 00:02 2023-07-20T01:32:53.501 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T01:32:53.745 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T01:32:53.874 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:32:53.982 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:32:53.994 INFO:teuthology.orchestra.run.smithi086.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 1.9 MB/s | 1.7 MB 00:00 2023-07-20T01:32:54.123 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:32:54.149 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T01:32:54.207 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:32:54.365 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T01:32:54.456 INFO:teuthology.orchestra.run.smithi086.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 526 kB/s | 241 kB 00:00 2023-07-20T01:32:54.511 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T01:32:54.520 INFO:teuthology.orchestra.run.smithi086.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 712 kB/s | 47 kB 00:00 2023-07-20T01:32:54.586 INFO:teuthology.orchestra.run.smithi086.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 219 kB/s | 15 kB 00:00 2023-07-20T01:32:54.618 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T01:32:54.657 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T01:32:54.737 INFO:teuthology.orchestra.run.smithi086.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 1.7 MB/s | 258 kB 00:00 2023-07-20T01:32:54.837 INFO:teuthology.orchestra.run.smithi086.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 1.6 MB/s | 165 kB 00:00 2023-07-20T01:32:54.892 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T01:32:54.895 INFO:teuthology.orchestra.run.smithi086.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 308 kB/s | 18 kB 00:00 2023-07-20T01:32:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T01:32:55.298 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T01:32:55.329 INFO:teuthology.orchestra.run.smithi086.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 707 kB/s | 306 kB 00:00 2023-07-20T01:32:55.430 INFO:teuthology.orchestra.run.smithi086.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 588 kB/s | 59 kB 00:00 2023-07-20T01:32:55.449 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T01:32:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T01:32:55.513 INFO:teuthology.orchestra.run.smithi086.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 901 kB/s | 75 kB 00:00 2023-07-20T01:32:55.638 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T01:32:55.816 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T01:32:55.830 INFO:teuthology.orchestra.run.smithi086.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.5 MB/s | 476 kB 00:00 2023-07-20T01:32:55.922 INFO:teuthology.orchestra.run.smithi086.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 476 kB/s | 44 kB 00:00 2023-07-20T01:32:55.958 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T01:32:56.039 INFO:teuthology.orchestra.run.smithi086.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.6 MB/s | 185 kB 00:00 2023-07-20T01:32:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T01:32:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T01:32:56.117 INFO:teuthology.orchestra.run.smithi086.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 440 kB/s | 33 kB 00:00 2023-07-20T01:32:56.190 INFO:teuthology.orchestra.run.smithi086.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 217 kB/s | 16 kB 00:00 2023-07-20T01:32:56.229 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T01:32:56.306 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T01:32:56.343 INFO:teuthology.orchestra.run.smithi086.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2023-07-20T01:32:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T01:32:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T01:32:56.490 INFO:teuthology.orchestra.run.smithi086.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.3 MB/s | 384 kB 00:00 2023-07-20T01:32:56.574 INFO:teuthology.orchestra.run.smithi086.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 977 kB/s | 81 kB 00:00 2023-07-20T01:32:56.641 INFO:teuthology.orchestra.run.smithi086.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 167 kB/s | 11 kB 00:00 2023-07-20T01:32:56.693 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T01:32:56.707 INFO:teuthology.orchestra.run.smithi086.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 263 kB/s | 18 kB 00:00 2023-07-20T01:32:56.932 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T01:32:57.050 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T01:32:57.058 INFO:teuthology.orchestra.run.smithi086.stdout:(50/131): ceph-mgr-diskprediction-local-18.0.0- 2.0 MB/s | 7.4 MB 00:03 2023-07-20T01:32:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T01:32:57.100 INFO:teuthology.orchestra.run.smithi086.stdout:(51/131): python3-kubernetes-11.0.0-6.el8.noarc 3.0 MB/s | 1.2 MB 00:00 2023-07-20T01:32:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T01:32:57.167 INFO:teuthology.orchestra.run.smithi086.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 884 kB/s | 59 kB 00:00 2023-07-20T01:32:57.234 INFO:teuthology.orchestra.run.smithi086.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 905 kB/s | 60 kB 00:00 2023-07-20T01:32:57.259 INFO:teuthology.orchestra.run.smithi086.stdout:(54/131): python3-logutils-0.3.5-11.el8.noarch. 243 kB/s | 49 kB 00:00 2023-07-20T01:32:57.313 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-20T01:32:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T01:32:57.418 INFO:teuthology.orchestra.run.smithi086.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2023-07-20T01:32:57.485 INFO:teuthology.orchestra.run.smithi086.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 505 kB/s | 34 kB 00:00 2023-07-20T01:32:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T01:32:57.602 INFO:teuthology.orchestra.run.smithi086.stdout:(57/131): python3-portend-2.6-1.el8.noarch.rpm 45 kB/s | 16 kB 00:00 2023-07-20T01:32:57.660 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T01:32:57.686 INFO:teuthology.orchestra.run.smithi086.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 981 kB/s | 196 kB 00:00 2023-07-20T01:32:57.699 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-20T01:32:57.777 INFO:teuthology.orchestra.run.smithi086.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 204 kB/s | 19 kB 00:00 2023-07-20T01:32:57.802 INFO:teuthology.orchestra.run.smithi086.stdout:(60/131): python3-rsa-4.8-1.el8.noarch.rpm 307 kB/s | 61 kB 00:00 2023-07-20T01:32:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T01:32:57.877 INFO:teuthology.orchestra.run.smithi086.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 390 kB/s | 29 kB 00:00 2023-07-20T01:32:57.919 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-20T01:32:57.953 INFO:teuthology.orchestra.run.smithi086.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 367 kB/s | 27 kB 00:00 2023-07-20T01:32:57.986 INFO:teuthology.orchestra.run.smithi086.stdout:(63/131): python3-singledispatch-3.4.0.3-18.el8 114 kB/s | 24 kB 00:00 2023-07-20T01:32:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T01:32:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-20T01:32:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T01:32:58.220 INFO:teuthology.orchestra.run.smithi086.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.1 MB/s | 251 kB 00:00 2023-07-20T01:32:58.320 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T01:32:58.343 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-20T01:32:58.362 INFO:teuthology.orchestra.run.smithi086.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 624 kB/s | 255 kB 00:00 2023-07-20T01:32:58.505 INFO:teuthology.orchestra.run.smithi086.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 603 kB/s | 85 kB 00:00 2023-07-20T01:32:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T01:32:58.521 INFO:teuthology.orchestra.run.smithi086.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 204 kB/s | 61 kB 00:00 2023-07-20T01:32:58.554 INFO:teuthology.orchestra.run.smithi086.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 458 kB/s | 23 kB 00:00 2023-07-20T01:32:58.562 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T01:32:58.580 INFO:teuthology.orchestra.run.smithi086.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T01:32:58.630 INFO:teuthology.orchestra.run.smithi086.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 940 kB/s | 47 kB 00:00 2023-07-20T01:32:58.672 INFO:teuthology.orchestra.run.smithi086.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-07-20T01:32:58.706 INFO:teuthology.orchestra.run.smithi086.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2023-07-20T01:32:58.730 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T01:32:58.747 INFO:teuthology.orchestra.run.smithi086.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-07-20T01:32:58.790 INFO:teuthology.orchestra.run.smithi086.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.1 MB/s | 48 kB 00:00 2023-07-20T01:32:58.831 INFO:teuthology.orchestra.run.smithi086.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-07-20T01:32:58.873 INFO:teuthology.orchestra.run.smithi086.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.0 MB/s | 43 kB 00:00 2023-07-20T01:32:58.881 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T01:32:58.894 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T01:32:58.915 INFO:teuthology.orchestra.run.smithi086.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 2.6 MB/s | 109 kB 00:00 2023-07-20T01:32:58.957 INFO:teuthology.orchestra.run.smithi086.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2023-07-20T01:32:59.016 INFO:teuthology.orchestra.run.smithi086.stdout:(79/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.4 MB/s | 560 kB 00:00 2023-07-20T01:32:59.066 INFO:teuthology.orchestra.run.smithi086.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 372 kB/s | 19 kB 00:00 2023-07-20T01:32:59.108 INFO:teuthology.orchestra.run.smithi086.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.6 MB/s | 155 kB 00:00 2023-07-20T01:32:59.159 INFO:teuthology.orchestra.run.smithi086.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 11 MB/s | 559 kB 00:00 2023-07-20T01:32:59.201 INFO:teuthology.orchestra.run.smithi086.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 429 kB/s | 18 kB 00:00 2023-07-20T01:32:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T01:32:59.251 INFO:teuthology.orchestra.run.smithi086.stdout:(84/131): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2023-07-20T01:32:59.293 INFO:teuthology.orchestra.run.smithi086.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.6 MB/s | 238 kB 00:00 2023-07-20T01:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 28 MB/s | 2.3 MB 00:00 2023-07-20T01:32:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-07-20T01:32:59.422 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T01:32:59.460 INFO:teuthology.orchestra.run.smithi086.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2023-07-20T01:32:59.494 INFO:teuthology.orchestra.run.smithi086.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 1.8 MB/s | 60 kB 00:00 2023-07-20T01:32:59.536 INFO:teuthology.orchestra.run.smithi086.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 2.3 MB/s | 98 kB 00:00 2023-07-20T01:32:59.578 INFO:teuthology.orchestra.run.smithi086.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2023-07-20T01:32:59.611 INFO:teuthology.orchestra.run.smithi086.stdout:(92/131): libcom_err-devel-1.45.6-4.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2023-07-20T01:32:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T01:32:59.664 INFO:teuthology.orchestra.run.smithi086.stdout:(93/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 13 MB/s | 644 kB 00:00 2023-07-20T01:32:59.704 INFO:teuthology.orchestra.run.smithi086.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 4.0 MB/s | 171 kB 00:00 2023-07-20T01:32:59.747 INFO:teuthology.orchestra.run.smithi086.stdout:(95/131): NetworkManager-initscripts-updown-1.3 3.2 MB/s | 138 kB 00:00 2023-07-20T01:32:59.829 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T01:32:59.883 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T01:33:00.021 INFO:teuthology.orchestra.run.smithi086.stdout:(96/131): python3-scipy-1.0.0-21.module+el8.5.0 50 MB/s | 14 MB 00:00 2023-07-20T01:33:00.063 INFO:teuthology.orchestra.run.smithi086.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2023-07-20T01:33:00.097 INFO:teuthology.orchestra.run.smithi086.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2023-07-20T01:33:00.138 INFO:teuthology.orchestra.run.smithi086.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-20T01:33:00.163 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-20T01:33:00.180 INFO:teuthology.orchestra.run.smithi086.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-07-20T01:33:00.214 INFO:teuthology.orchestra.run.smithi086.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 1.1 MB/s | 39 kB 00:00 2023-07-20T01:33:00.224 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T01:33:00.417 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T01:33:00.446 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-20T01:33:00.548 INFO:teuthology.orchestra.run.smithi086.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 2.0 MB/s | 688 kB 00:00 2023-07-20T01:33:00.598 INFO:teuthology.orchestra.run.smithi086.stdout:(103/131): python3-jinja2-2.10.1-3.el8.noarch.r 11 MB/s | 538 kB 00:00 2023-07-20T01:33:00.611 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T01:33:00.640 INFO:teuthology.orchestra.run.smithi086.stdout:(104/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-07-20T01:33:00.682 INFO:teuthology.orchestra.run.smithi086.stdout:(105/131): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-07-20T01:33:00.703 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-20T01:33:00.732 INFO:teuthology.orchestra.run.smithi086.stdout:(106/131): automake-1.16.1-7.el8.noarch.rpm 14 MB/s | 713 kB 00:00 2023-07-20T01:33:00.849 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T01:33:00.857 INFO:teuthology.orchestra.run.smithi086.stdout:(107/131): python3-babel-2.5.1-7.el8.noarch.rpm 38 MB/s | 4.8 MB 00:00 2023-07-20T01:33:00.900 INFO:teuthology.orchestra.run.smithi086.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 636 kB/s | 27 kB 00:00 2023-07-20T01:33:00.923 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T01:33:00.942 INFO:teuthology.orchestra.run.smithi086.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-07-20T01:33:00.984 INFO:teuthology.orchestra.run.smithi086.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.0 MB/s | 44 kB 00:00 2023-07-20T01:33:01.006 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T01:33:01.084 INFO:teuthology.orchestra.run.smithi086.stdout:(111/131): python3-numpy-1.14.3-10.el8.x86_64.r 37 MB/s | 3.7 MB 00:00 2023-07-20T01:33:01.134 INFO:teuthology.orchestra.run.smithi086.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 9.0 MB/s | 457 kB 00:00 2023-07-20T01:33:01.176 INFO:teuthology.orchestra.run.smithi086.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 5.3 MB/s | 225 kB 00:00 2023-07-20T01:33:01.180 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T01:33:01.218 INFO:teuthology.orchestra.run.smithi086.stdout:(114/131): uuid-1.6.2-43.el8.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2023-07-20T01:33:01.252 INFO:teuthology.orchestra.run.smithi086.stdout:(115/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 3.7 MB/s | 126 kB 00:00 2023-07-20T01:33:01.294 INFO:teuthology.orchestra.run.smithi086.stdout:(116/131): python3-jsonpointer-1.10-11.el8.noar 480 kB/s | 20 kB 00:00 2023-07-20T01:33:01.335 INFO:teuthology.orchestra.run.smithi086.stdout:(117/131): thrift-0.13.0-2.el8.x86_64.rpm 631 kB/s | 1.7 MB 00:02 2023-07-20T01:33:01.352 INFO:teuthology.orchestra.run.smithi086.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.7 MB/s | 103 kB 00:00 2023-07-20T01:33:01.411 INFO:teuthology.orchestra.run.smithi086.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 9.2 MB/s | 709 kB 00:00 2023-07-20T01:33:01.470 INFO:teuthology.orchestra.run.smithi086.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 3.3 MB/s | 198 kB 00:00 2023-07-20T01:33:01.526 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T01:33:01.545 INFO:teuthology.orchestra.run.smithi086.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 26 MB/s | 4.9 MB 00:00 2023-07-20T01:33:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:(122/131): openblas-0.3.15-3.el8.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-07-20T01:33:01.712 INFO:teuthology.orchestra.run.smithi086.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2023-07-20T01:33:01.787 INFO:teuthology.orchestra.run.smithi086.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.2 MB/s | 354 kB 00:00 2023-07-20T01:33:01.805 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T01:33:01.862 INFO:teuthology.orchestra.run.smithi086.stdout:(125/131): postgresql-contrib-10.21-2.module+el 11 MB/s | 808 kB 00:00 2023-07-20T01:33:01.971 INFO:teuthology.orchestra.run.smithi086.stdout:(126/131): postgresql-server-10.21-2.module+el8 20 MB/s | 5.1 MB 00:00 2023-07-20T01:33:02.005 INFO:teuthology.orchestra.run.smithi086.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 11 MB/s | 1.5 MB 00:00 2023-07-20T01:33:02.042 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T01:33:02.269 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T01:33:02.450 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T01:33:02.630 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T01:33:02.801 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T01:33:02.884 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T01:33:03.093 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-20T01:33:03.116 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T01:33:03.279 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T01:33:03.311 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T01:33:03.471 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T01:33:03.510 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T01:33:03.655 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T01:33:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T01:33:03.849 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T01:33:03.860 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T01:33:04.022 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T01:33:04.038 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-20T01:33:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T01:33:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-20T01:33:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T01:33:04.400 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-20T01:33:04.553 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-20T01:33:04.583 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T01:33:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:(128/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 1.4 MB/s | 3.7 MB 00:02 2023-07-20T01:33:04.747 INFO:teuthology.orchestra.run.smithi086.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 12 MB/s | 798 kB 00:00 2023-07-20T01:33:04.814 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T01:33:04.916 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-20T01:33:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T01:33:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-20T01:33:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:(130/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 1.1 MB/s | 3.6 MB 00:03 2023-07-20T01:33:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T01:33:05.474 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T01:33:05.683 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T01:33:05.771 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-20T01:33:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T01:33:06.051 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-20T01:33:06.258 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-20T01:33:06.480 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-20T01:33:06.745 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-20T01:33:07.040 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-20T01:33:07.068 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T01:33:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-20T01:33:07.331 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T01:33:07.415 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-20T01:33:07.551 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T01:33:07.771 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-20T01:33:07.861 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T01:33:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-20T01:33:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T01:33:08.135 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-20T01:33:08.312 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T01:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-20T01:33:08.727 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-20T01:33:08.971 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-20T01:33:09.119 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T01:33:09.162 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-20T01:33:09.290 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:33:09.369 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:33:09.370 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-20T01:33:09.532 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T01:33:09.536 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-20T01:33:09.721 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T01:33:09.795 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T01:33:10.064 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-20T01:33:10.359 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-20T01:33:10.529 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T01:33:10.599 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T01:33:10.813 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-20T01:33:10.965 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-20T01:33:11.105 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T01:33:11.295 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T01:33:11.558 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T01:33:11.820 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T01:33:11.968 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T01:33:12.117 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T01:33:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T01:33:12.536 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T01:33:12.718 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T01:33:12.882 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T01:33:13.019 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T01:33:13.277 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T01:33:13.387 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:33:13.525 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T01:33:13.574 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:33:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T01:33:13.974 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T01:33:14.187 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T01:33:14.189 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:33:14.354 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T01:33:14.482 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:33:14.563 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:33:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T01:33:14.690 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:33:15.038 INFO:teuthology.orchestra.run.smithi086.stdout:(131/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8. 1.6 MB/s | 48 MB 00:30 2023-07-20T01:33:15.041 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:33:15.041 INFO:teuthology.orchestra.run.smithi086.stdout:Total 4.7 MB/s | 206 MB 00:44 2023-07-20T01:33:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T01:33:15.888 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-07-20T01:33:16.007 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-07-20T01:33:16.007 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-07-20T01:33:16.093 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T01:33:16.313 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T01:33:16.656 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T01:33:16.912 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T01:33:17.130 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-07-20T01:33:17.134 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-07-20T01:33:17.141 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T01:33:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T01:33:18.095 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:33:18.158 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:33:18.345 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T01:33:21.930 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-07-20T01:33:22.115 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T01:33:22.260 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:33:22.332 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T01:33:22.461 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:33:22.509 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T01:33:22.605 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T01:33:22.809 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T01:33:23.076 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:33:23.214 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T01:33:23.357 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:33:23.425 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T01:33:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:33:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:33:23.822 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T01:33:24.000 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:33:24.148 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:33:24.316 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:33:24.416 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:33:24.572 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:33:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T01:33:24.962 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T01:33:25.186 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T01:33:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T01:33:25.762 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T01:33:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:33:26.030 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:33:26.158 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:33:26.306 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:33:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T01:33:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T01:33:26.898 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T01:33:27.033 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T01:33:27.172 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T01:33:27.316 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T01:33:27.932 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T01:33:28.083 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T01:33:28.680 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T01:33:28.833 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T01:33:29.525 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T01:33:29.773 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T01:33:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T01:33:30.184 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T01:33:30.381 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T01:33:30.503 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T01:33:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T01:33:30.899 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T01:33:32.653 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T01:33:33.661 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T01:33:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T01:33:34.251 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T01:33:34.269 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:33:34.269 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /sys 2023-07-20T01:33:34.270 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /proc 2023-07-20T01:33:34.270 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /dev 2023-07-20T01:33:34.270 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /run 2023-07-20T01:33:34.270 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /mnt 2023-07-20T01:33:34.270 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /var/tmp 2023-07-20T01:33:34.270 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /home 2023-07-20T01:33:34.270 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /tmp 2023-07-20T01:33:34.271 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /dev 2023-07-20T01:33:34.271 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:33:34.472 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T01:33:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:33:34.668 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T01:33:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T01:33:35.046 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T01:33:35.180 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T01:33:35.314 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T01:33:35.450 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T01:33:35.615 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T01:33:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T01:33:35.944 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T01:33:36.160 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T01:33:36.305 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:33:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T01:33:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T01:33:36.479 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:33:36.586 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T01:33:36.710 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:33:36.744 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T01:33:36.878 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:33:36.897 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T01:33:37.066 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T01:33:37.226 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T01:33:37.360 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T01:33:37.360 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:33:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:33:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:33:37.572 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T01:33:37.770 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T01:33:38.127 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T01:33:38.441 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T01:33:38.684 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T01:33:38.895 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T01:33:39.120 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T01:33:39.311 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:33:39.423 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T01:33:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:33:39.625 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T01:33:39.631 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T01:33:39.818 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T01:33:39.824 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T01:33:40.027 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:33:40.037 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T01:33:40.228 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T01:33:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:33:40.394 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T01:33:40.486 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:33:40.692 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T01:33:40.936 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T01:33:41.194 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T01:33:41.407 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T01:33:41.510 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:33:41.555 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T01:33:41.669 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:33:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:33:41.894 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T01:33:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T01:33:42.296 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T01:33:42.356 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:33:42.392 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:33:42.530 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T01:33:42.535 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:33:42.546 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:33:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T01:33:42.836 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T01:33:42.942 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:33:42.942 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2023-07-20T01:33:42.942 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2023-07-20T01:33:42.943 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2023-07-20T01:33:42.943 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /run 2023-07-20T01:33:42.943 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2023-07-20T01:33:42.943 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2023-07-20T01:33:42.943 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2023-07-20T01:33:42.943 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2023-07-20T01:33:42.943 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2023-07-20T01:33:42.944 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:33:43.058 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T01:33:43.260 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T01:33:43.307 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:33:43.443 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T01:33:43.609 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T01:33:43.745 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T01:33:43.913 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T01:33:43.976 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T01:33:44.080 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:33:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:33:44.193 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T01:33:44.252 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:33:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:33:44.378 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T01:33:44.468 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:33:44.603 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T01:33:44.830 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T01:33:44.842 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:33:44.979 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T01:33:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:33:44.995 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:33:45.187 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:33:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T01:33:45.418 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:33:45.582 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:33:46.064 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:33:46.198 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:33:46.216 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:33:46.267 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T01:33:46.517 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T01:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T01:33:47.065 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T01:33:47.346 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T01:33:47.582 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T01:33:47.991 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:33:48.172 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:33:48.288 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T01:33:48.346 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T01:33:48.505 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:33:48.532 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T01:33:48.578 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:33:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:33:48.738 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T01:33:48.946 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:33:49.244 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:33:49.861 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:33:50.270 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:33:50.414 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:33:50.510 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:33:50.804 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T01:33:50.898 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T01:33:51.095 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T01:33:51.126 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:33:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:33:51.270 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T01:33:51.314 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:33:51.326 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:33:51.493 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T01:33:51.650 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T01:33:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T01:33:52.079 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T01:33:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T01:33:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T01:33:52.430 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:33:52.732 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T01:33:52.764 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:33:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:33:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:33:53.106 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:33:53.150 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:33:53.289 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:33:53.344 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:33:53.601 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:33:53.664 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:33:53.690 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:33:53.789 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:33:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:33:54.730 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:33:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:33:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:33:55.142 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T01:33:55.408 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T01:33:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T01:33:56.034 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T01:33:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T01:33:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T01:33:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T01:33:56.495 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T01:33:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T01:33:56.691 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T01:33:56.818 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T01:33:56.819 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:33:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:33:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:33:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:33:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:33:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:33:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T01:33:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T01:33:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T01:33:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T01:33:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T01:33:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:33:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:33:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:33:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T01:33:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T01:33:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T01:33:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T01:33:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T01:33:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T01:33:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T01:33:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T01:33:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T01:33:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T01:33:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T01:33:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T01:33:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T01:33:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T01:33:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T01:33:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T01:33:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T01:33:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T01:33:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T01:33:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T01:33:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T01:33:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T01:33:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T01:33:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T01:33:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T01:33:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T01:33:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T01:33:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T01:33:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T01:33:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T01:33:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T01:33:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T01:33:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T01:33:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T01:33:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T01:33:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T01:33:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T01:33:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T01:33:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T01:33:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T01:33:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T01:33:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T01:33:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T01:33:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T01:33:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T01:33:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T01:33:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T01:33:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T01:33:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T01:33:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T01:33:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T01:33:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T01:33:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T01:33:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T01:33:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T01:33:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T01:33:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T01:33:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T01:33:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T01:33:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T01:33:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T01:33:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T01:33:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T01:33:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T01:33:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T01:33:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T01:33:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T01:33:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T01:33:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T01:33:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T01:33:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T01:33:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T01:33:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T01:33:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T01:33:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T01:33:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T01:33:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T01:33:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T01:33:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T01:33:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T01:33:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T01:33:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T01:33:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T01:33:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T01:33:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T01:33:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T01:33:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T01:33:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T01:33:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T01:33:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T01:33:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T01:33:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T01:33:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T01:33:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T01:33:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T01:33:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T01:33:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T01:33:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T01:33:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T01:33:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T01:33:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T01:33:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T01:33:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T01:33:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T01:33:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T01:33:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T01:33:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T01:33:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T01:33:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T01:33:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T01:33:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T01:33:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T01:33:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T01:33:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T01:33:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T01:33:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T01:33:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T01:33:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T01:33:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T01:33:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T01:33:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T01:33:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T01:33:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T01:33:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:33:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T01:33:58.647 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:33:58.955 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T01:33:59.121 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T01:33:59.349 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T01:34:00.257 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-20T01:34:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:34:00.445 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:22 ago on Thu 20 Jul 2023 01:31:38 AM UTC. 2023-07-20T01:34:00.599 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-20T01:34:00.758 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 119/134 2023-07-20T01:34:00.909 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T01:34:01.126 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T01:34:01.289 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T01:34:03.651 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:34:03.895 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:34:03.914 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:34:04.071 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T01:34:04.413 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T01:34:04.735 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T01:34:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T01:34:05.195 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T01:34:05.205 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T01:34:05.382 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-20T01:34:05.553 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T01:34:05.642 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T01:34:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T01:34:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T01:34:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:34:06.090 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:34:06.090 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:34:06.327 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:34:06.528 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:34:07.049 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:34:07.049 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T01:34:07.049 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T01:34:07.049 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T01:34:07.050 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T01:34:07.050 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T01:34:07.050 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:34:07.050 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:34:07.051 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:34:07.051 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T01:34:07.051 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T01:34:07.051 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T01:34:07.052 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T01:34:07.052 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T01:34:07.052 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T01:34:07.053 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T01:34:07.053 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T01:34:07.053 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T01:34:07.054 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T01:34:07.054 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T01:34:07.054 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T01:34:07.056 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T01:34:07.057 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T01:34:07.057 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T01:34:07.057 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T01:34:07.058 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T01:34:07.058 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T01:34:07.058 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T01:34:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T01:34:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T01:34:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T01:34:07.070 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T01:34:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T01:34:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T01:34:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T01:34:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T01:34:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T01:34:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T01:34:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T01:34:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T01:34:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T01:34:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T01:34:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T01:34:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T01:34:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T01:34:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T01:34:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T01:34:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T01:34:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T01:34:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T01:34:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T01:34:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T01:34:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T01:34:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T01:34:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T01:34:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T01:34:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T01:34:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T01:34:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T01:34:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T01:34:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T01:34:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T01:34:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T01:34:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T01:34:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T01:34:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T01:34:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T01:34:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T01:34:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T01:34:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T01:34:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T01:34:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T01:34:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-20T01:34:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-20T01:34:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-20T01:34:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-20T01:34:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-20T01:34:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-20T01:34:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-20T01:34:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-20T01:34:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-20T01:34:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-20T01:34:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-20T01:34:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-20T01:34:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-20T01:34:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-20T01:34:07.081 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-20T01:34:07.081 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-20T01:34:07.081 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-20T01:34:07.081 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-20T01:34:07.081 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-20T01:34:07.081 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-20T01:34:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-20T01:34:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-20T01:34:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-20T01:34:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-20T01:34:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-20T01:34:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-20T01:34:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-20T01:34:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-20T01:34:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-20T01:34:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-20T01:34:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-20T01:34:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-20T01:34:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-20T01:34:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-20T01:34:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-20T01:34:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-20T01:34:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-20T01:34:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-20T01:34:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-20T01:34:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-20T01:34:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-20T01:34:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-20T01:34:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-20T01:34:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-20T01:34:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-20T01:34:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-20T01:34:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-20T01:34:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-20T01:34:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T01:34:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-20T01:34:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T01:34:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-20T01:34:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-20T01:34:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-20T01:34:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-20T01:34:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-20T01:34:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T01:34:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T01:34:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T01:34:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T01:34:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:34:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T01:34:09.548 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:34:09.549 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:19 ago on Thu 20 Jul 2023 01:31:50 AM UTC. 2023-07-20T01:34:09.651 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2023-07-20T01:34:09.885 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:34:09.886 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2023-07-20T01:34:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T01:34:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.886 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2023-07-20T01:34:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T01:34:09.886 INFO:teuthology.orchestra.run.smithi043.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T01:34:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T01:34:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:09.887 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:09.888 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:09.889 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T01:34:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T01:34:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T01:34:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T01:34:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: jq-1.6-3.el8.x86_64 2023-07-20T01:34:09.890 INFO:teuthology.orchestra.run.smithi043.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T01:34:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T01:34:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T01:34:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T01:34:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:34:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T01:34:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T01:34:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:34:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T01:34:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.892 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T01:34:09.893 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.894 INFO:teuthology.orchestra.run.smithi043.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T01:34:09.894 INFO:teuthology.orchestra.run.smithi043.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T01:34:09.895 INFO:teuthology.orchestra.run.smithi043.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T01:34:09.895 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T01:34:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T01:34:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T01:34:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T01:34:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T01:34:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T01:34:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T01:34:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T01:34:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T01:34:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T01:34:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T01:34:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:34:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:34:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:34:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T01:34:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T01:34:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T01:34:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T01:34:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T01:34:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T01:34:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T01:34:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T01:34:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T01:34:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T01:34:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T01:34:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T01:34:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T01:34:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T01:34:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T01:34:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T01:34:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T01:34:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T01:34:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T01:34:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T01:34:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T01:34:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T01:34:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T01:34:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T01:34:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T01:34:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T01:34:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T01:34:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T01:34:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T01:34:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T01:34:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T01:34:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T01:34:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T01:34:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T01:34:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T01:34:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T01:34:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T01:34:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T01:34:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T01:34:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T01:34:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T01:34:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T01:34:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T01:34:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T01:34:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T01:34:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T01:34:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T01:34:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T01:34:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T01:34:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T01:34:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T01:34:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T01:34:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T01:34:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T01:34:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T01:34:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: 2023-07-20T01:34:09.907 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2023-07-20T01:34:10.691 DEBUG:teuthology.parallel:result is None 2023-07-20T01:34:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:34:12.682 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2023-07-20T01:34:12.682 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2023-07-20T01:34:12.682 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2023-07-20T01:34:12.682 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /run 2023-07-20T01:34:12.682 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2023-07-20T01:34:12.682 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2023-07-20T01:34:12.682 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2023-07-20T01:34:12.682 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2023-07-20T01:34:12.683 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2023-07-20T01:34:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:34:13.060 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:34:14.640 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:34:14.823 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:34:15.041 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:34:15.218 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:34:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:34:15.809 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:34:15.826 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:34:16.001 INFO:teuthology.orchestra.run.smithi003.stdout:Installed products updated. 2023-07-20T01:34:16.233 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:34:16.233 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2023-07-20T01:34:16.233 INFO:teuthology.orchestra.run.smithi003.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T01:34:16.234 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.234 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.234 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-07-20T01:34:16.234 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T01:34:16.234 INFO:teuthology.orchestra.run.smithi003.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T01:34:16.234 INFO:teuthology.orchestra.run.smithi003.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T01:34:16.234 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.234 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.235 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.235 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.235 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:16.235 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.235 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.236 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.236 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:16.236 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:16.236 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:16.236 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:16.236 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:16.236 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.237 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.237 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:16.237 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.237 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.237 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.237 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:16.237 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:16.237 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.238 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:16.238 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T01:34:16.238 INFO:teuthology.orchestra.run.smithi003.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T01:34:16.238 INFO:teuthology.orchestra.run.smithi003.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T01:34:16.238 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T01:34:16.238 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-3.el8.x86_64 2023-07-20T01:34:16.238 INFO:teuthology.orchestra.run.smithi003.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T01:34:16.239 INFO:teuthology.orchestra.run.smithi003.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T01:34:16.239 INFO:teuthology.orchestra.run.smithi003.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T01:34:16.239 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.239 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.239 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.239 INFO:teuthology.orchestra.run.smithi003.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T01:34:16.239 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:34:16.239 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T01:34:16.240 INFO:teuthology.orchestra.run.smithi003.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T01:34:16.240 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:34:16.240 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T01:34:16.240 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.240 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.240 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T01:34:16.240 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.240 INFO:teuthology.orchestra.run.smithi003.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T01:34:16.241 INFO:teuthology.orchestra.run.smithi003.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T01:34:16.241 INFO:teuthology.orchestra.run.smithi003.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T01:34:16.241 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T01:34:16.241 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T01:34:16.241 INFO:teuthology.orchestra.run.smithi003.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T01:34:16.241 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T01:34:16.241 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T01:34:16.242 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T01:34:16.242 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T01:34:16.242 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T01:34:16.242 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T01:34:16.242 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T01:34:16.242 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T01:34:16.242 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:34:16.242 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:34:16.243 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:34:16.243 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T01:34:16.243 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T01:34:16.243 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T01:34:16.243 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T01:34:16.243 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T01:34:16.243 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.244 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.244 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.244 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T01:34:16.244 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T01:34:16.244 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T01:34:16.244 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T01:34:16.244 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T01:34:16.244 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T01:34:16.245 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T01:34:16.245 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T01:34:16.245 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T01:34:16.245 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T01:34:16.245 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T01:34:16.245 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T01:34:16.245 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T01:34:16.245 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T01:34:16.246 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T01:34:16.246 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T01:34:16.246 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T01:34:16.246 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T01:34:16.246 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T01:34:16.246 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T01:34:16.246 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T01:34:16.246 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T01:34:16.247 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T01:34:16.247 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T01:34:16.247 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T01:34:16.247 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T01:34:16.247 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T01:34:16.248 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T01:34:16.248 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T01:34:16.248 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.248 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.248 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T01:34:16.248 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T01:34:16.248 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.248 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T01:34:16.249 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T01:34:16.249 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T01:34:16.249 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T01:34:16.249 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T01:34:16.249 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T01:34:16.249 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T01:34:16.249 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T01:34:16.250 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T01:34:16.250 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T01:34:16.250 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T01:34:16.250 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T01:34:16.250 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T01:34:16.250 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.250 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.250 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:16.251 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T01:34:16.251 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T01:34:16.251 INFO:teuthology.orchestra.run.smithi003.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T01:34:16.251 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T01:34:16.251 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T01:34:16.252 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T01:34:16.252 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:34:16.252 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-07-20T01:34:16.555 DEBUG:teuthology.parallel:result is None 2023-07-20T01:34:17.497 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:34:17.692 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:34:17.807 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T01:34:18.016 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T01:34:18.184 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:34:18.401 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:34:18.672 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:34:19.635 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:34:19.800 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:34:19.893 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:34:20.283 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T01:34:20.497 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:34:20.534 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:34:20.684 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:34:20.695 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:34:22.009 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T01:34:22.194 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:34:22.228 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:34:22.359 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:34:22.401 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:34:22.534 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:34:22.886 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:34:23.009 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:34:27.550 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:34:28.411 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T01:34:28.722 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T01:34:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T01:34:29.122 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T01:34:29.282 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T01:34:29.478 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T01:34:29.712 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T01:34:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T01:34:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T01:34:32.213 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:34:32.449 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:34:32.468 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:34:32.599 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T01:34:32.907 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T01:34:33.199 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T01:34:33.523 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T01:34:33.707 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T01:34:33.787 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T01:34:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T01:34:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T01:34:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T01:34:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T01:34:34.292 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T01:34:34.293 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:34:34.443 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:34:34.443 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:34:34.662 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:34:34.855 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:34:35.361 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:34:35.361 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T01:34:35.361 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T01:34:35.362 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T01:34:35.362 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T01:34:35.362 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T01:34:35.362 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:34:35.362 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:34:35.362 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:34:35.362 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T01:34:35.363 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T01:34:35.363 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T01:34:35.363 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T01:34:35.363 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T01:34:35.363 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T01:34:35.363 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T01:34:35.363 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T01:34:35.364 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T01:34:35.364 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T01:34:35.364 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T01:34:35.364 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T01:34:35.364 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T01:34:35.364 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T01:34:35.364 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T01:34:35.365 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T01:34:35.365 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T01:34:35.365 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T01:34:35.365 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T01:34:35.365 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T01:34:35.365 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T01:34:35.365 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T01:34:35.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T01:34:35.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T01:34:35.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T01:34:35.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T01:34:35.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T01:34:35.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T01:34:35.374 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T01:34:35.374 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T01:34:35.374 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T01:34:35.374 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T01:34:35.374 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T01:34:35.374 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T01:34:35.374 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T01:34:35.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T01:34:35.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T01:34:35.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T01:34:35.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T01:34:35.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T01:34:35.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T01:34:35.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T01:34:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T01:34:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T01:34:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T01:34:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T01:34:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T01:34:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T01:34:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T01:34:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T01:34:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T01:34:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T01:34:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T01:34:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T01:34:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T01:34:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T01:34:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T01:34:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T01:34:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T01:34:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T01:34:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T01:34:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T01:34:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T01:34:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T01:34:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T01:34:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T01:34:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T01:34:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T01:34:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T01:34:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T01:34:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T01:34:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T01:34:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T01:34:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T01:34:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T01:34:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T01:34:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T01:34:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T01:34:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T01:34:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T01:34:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T01:34:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T01:34:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T01:34:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T01:34:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T01:34:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T01:34:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T01:34:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T01:34:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T01:34:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T01:34:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T01:34:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T01:34:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T01:34:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T01:34:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T01:34:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T01:34:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T01:34:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T01:34:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T01:34:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T01:34:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T01:34:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T01:34:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T01:34:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T01:34:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T01:34:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T01:34:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T01:34:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T01:34:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T01:34:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T01:34:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T01:34:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T01:34:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T01:34:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T01:34:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T01:34:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T01:34:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T01:34:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T01:34:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T01:34:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T01:34:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T01:34:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T01:34:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T01:34:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:34:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T01:34:37.789 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:34:37.789 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:18 ago on Thu 20 Jul 2023 01:32:19 AM UTC. 2023-07-20T01:34:42.360 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-07-20T01:34:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:34:42.572 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2023-07-20T01:34:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T01:34:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.573 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-07-20T01:34:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T01:34:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T01:34:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T01:34:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.573 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:42.574 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.575 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:34:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T01:34:42.576 INFO:teuthology.orchestra.run.smithi086.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T01:34:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T01:34:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T01:34:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: jq-1.6-3.el8.x86_64 2023-07-20T01:34:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T01:34:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T01:34:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T01:34:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.577 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T01:34:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:34:42.578 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T01:34:42.579 INFO:teuthology.orchestra.run.smithi086.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T01:34:42.579 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:34:42.579 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T01:34:42.579 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.579 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T01:34:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T01:34:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T01:34:42.580 INFO:teuthology.orchestra.run.smithi086.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T01:34:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T01:34:42.581 INFO:teuthology.orchestra.run.smithi086.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T01:34:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T01:34:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T01:34:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T01:34:42.582 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T01:34:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T01:34:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T01:34:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T01:34:42.583 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T01:34:42.584 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T01:34:42.584 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:34:42.584 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:34:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:34:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T01:34:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T01:34:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T01:34:42.585 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T01:34:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T01:34:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T01:34:42.586 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T01:34:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T01:34:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T01:34:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T01:34:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T01:34:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T01:34:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T01:34:42.587 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T01:34:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T01:34:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T01:34:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T01:34:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T01:34:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T01:34:42.588 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T01:34:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T01:34:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T01:34:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T01:34:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T01:34:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T01:34:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T01:34:42.589 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T01:34:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T01:34:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T01:34:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T01:34:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T01:34:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T01:34:42.590 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T01:34:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T01:34:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T01:34:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T01:34:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T01:34:42.591 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T01:34:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T01:34:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T01:34:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T01:34:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T01:34:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T01:34:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T01:34:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T01:34:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T01:34:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T01:34:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T01:34:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T01:34:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.593 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:34:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T01:34:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T01:34:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T01:34:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T01:34:42.594 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T01:34:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T01:34:42.595 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:34:42.595 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-07-20T01:34:43.142 DEBUG:teuthology.parallel:result is None 2023-07-20T01:34:43.143 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:34:43.409 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T01:34:43.438 INFO:teuthology.orchestra.run.smithi003.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:34:43.439 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:34:43.440 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T01:34:43.440 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:34:43.702 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T01:34:43.732 INFO:teuthology.orchestra.run.smithi043.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:34:43.733 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:34:43.733 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T01:34:43.733 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:34:44.031 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T01:34:44.058 INFO:teuthology.orchestra.run.smithi086.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:34:44.059 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:34:44.059 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T01:34:44.059 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-20T01:34:44.060 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:34:44.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T01:34:44.110 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2023-07-20T01:34:44.111 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T01:34:44.161 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-07-20T01:34:44.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T01:34:44.207 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-20T01:34:44.208 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:34:44.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T01:34:44.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T01:34:44.315 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2023-07-20T01:34:44.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T01:34:44.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T01:34:44.419 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-07-20T01:34:44.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T01:34:44.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T01:34:44.522 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-20T01:34:44.523 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:34:44.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T01:34:44.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T01:34:44.629 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2023-07-20T01:34:44.630 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T01:34:44.663 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T01:34:44.734 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-07-20T01:34:44.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T01:34:44.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T01:34:44.837 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-20T01:34:44.837 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:34:44.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T01:34:44.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T01:34:44.942 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2023-07-20T01:34:44.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T01:34:44.975 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T01:34:45.045 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-07-20T01:34:45.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T01:34:45.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T01:34:45.148 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-20T01:34:45.237 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2023-07-20T01:34:45.238 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:34:45.248 INFO:tasks.cephadm:Cluster fsid is 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:34:45.249 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-20T01:34:45.249 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.3', 'mon.b': '172.21.15.43', 'mon.c': '172.21.15.86'} 2023-07-20T01:34:45.250 INFO:tasks.cephadm:First mon is mon.a on smithi003 2023-07-20T01:34:45.250 INFO:tasks.cephadm:First mgr is x 2023-07-20T01:34:45.250 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-20T01:34:45.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo hostname $(hostname -s) 2023-07-20T01:34:45.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo hostname $(hostname -s) 2023-07-20T01:34:45.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2023-07-20T01:34:45.353 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-20T01:34:45.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:34:45.723 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71562/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-5040-g1b0b2eb0', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-5040.g1b0b2eb0'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-19 22:37:13.930734', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230719.201220', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-07-20T01:34:46.093 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-pdonnell-testing-20230719.201220, sha1 1b0b2eb005a7c51d97a788b9900d511e1394b537 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:34:46.094 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm 2023-07-20T01:34:46.094 DEBUG:teuthology.orchestra.run.smithi003:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T01:34:52.755 INFO:teuthology.orchestra.run.smithi003.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 01:34 /home/ubuntu/cephtest/cephadm 2023-07-20T01:34:52.755 DEBUG:teuthology.orchestra.run.smithi043:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T01:34:55.018 INFO:teuthology.orchestra.run.smithi043.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 01:34 /home/ubuntu/cephtest/cephadm 2023-07-20T01:34:55.019 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T01:34:57.073 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 01:34 /home/ubuntu/cephtest/cephadm 2023-07-20T01:34:57.074 DEBUG:teuthology.orchestra.run.smithi003:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T01:34:57.097 DEBUG:teuthology.orchestra.run.smithi043:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T01:34:57.120 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T01:34:57.156 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 on all hosts... 2023-07-20T01:34:57.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T01:34:57.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T01:34:57.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T01:34:58.387 INFO:teuthology.orchestra.run.smithi043.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T01:34:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T01:34:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T01:35:26.060 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-07-20T01:35:26.060 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T01:35:26.060 INFO:teuthology.orchestra.run.smithi086.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T01:35:26.060 INFO:teuthology.orchestra.run.smithi086.stdout: "repo_digests": [ 2023-07-20T01:35:26.061 INFO:teuthology.orchestra.run.smithi086.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T01:35:26.061 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2023-07-20T01:35:26.061 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-07-20T01:35:26.597 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2023-07-20T01:35:26.597 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T01:35:26.597 INFO:teuthology.orchestra.run.smithi043.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T01:35:26.597 INFO:teuthology.orchestra.run.smithi043.stdout: "repo_digests": [ 2023-07-20T01:35:26.598 INFO:teuthology.orchestra.run.smithi043.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T01:35:26.598 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2023-07-20T01:35:26.598 INFO:teuthology.orchestra.run.smithi043.stdout:} 2023-07-20T01:35:29.484 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-07-20T01:35:29.484 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T01:35:29.484 INFO:teuthology.orchestra.run.smithi003.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T01:35:29.485 INFO:teuthology.orchestra.run.smithi003.stdout: "repo_digests": [ 2023-07-20T01:35:29.485 INFO:teuthology.orchestra.run.smithi003.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T01:35:29.485 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-07-20T01:35:29.485 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-07-20T01:35:29.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph 2023-07-20T01:35:29.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph 2023-07-20T01:35:29.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2023-07-20T01:35:29.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /etc/ceph 2023-07-20T01:35:29.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /etc/ceph 2023-07-20T01:35:29.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2023-07-20T01:35:29.698 INFO:tasks.cephadm:Writing seed config... 2023-07-20T01:35:29.699 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-20T01:35:29.699 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-20T01:35:29.700 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-20T01:35:29.700 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-20T01:35:29.700 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-20T01:35:29.701 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-07-20T01:35:29.701 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-20T01:35:29.701 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-20T01:35:29.701 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-20T01:35:29.701 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-20T01:35:29.702 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-20T01:35:29.702 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-20T01:35:29.702 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-20T01:35:29.702 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-20T01:35:29.702 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-20T01:35:29.703 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-20T01:35:29.703 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-20T01:35:29.703 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-20T01:35:29.703 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-20T01:35:29.703 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-20T01:35:29.704 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-20T01:35:29.704 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-20T01:35:29.704 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-20T01:35:29.704 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-20T01:35:29.704 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-20T01:35:29.705 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-20T01:35:29.705 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-20T01:35:29.705 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-07-20T01:35:29.705 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-07-20T01:35:29.705 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-07-20T01:35:29.705 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-20T01:35:29.706 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-20T01:35:29.706 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-07-20T01:35:29.706 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-07-20T01:35:29.706 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-20T01:35:29.706 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-20T01:35:29.707 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-07-20T01:35:29.707 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-20T01:35:29.707 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-20T01:35:29.707 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-20T01:35:29.707 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-20T01:35:29.708 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-20T01:35:29.708 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-20T01:35:29.708 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-20T01:35:29.708 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-07-20T01:35:29.709 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:35:29.709 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-20T01:35:29.726 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9b4fd2fc-269d-11ee-9b34-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-20T01:35:29.727 DEBUG:teuthology.orchestra.run.smithi003:mon.a> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.a.service 2023-07-20T01:35:29.769 DEBUG:teuthology.orchestra.run.smithi003:mgr.x> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.x.service 2023-07-20T01:35:29.795 INFO:journalctl@ceph.mon.a.smithi003.stdout:-- Logs begin at Thu 2023-07-20 01:20:47 UTC. -- 2023-07-20T01:35:29.796 INFO:tasks.cephadm:Bootstrapping... 2023-07-20T01:35:29.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 -v bootstrap --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.3 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:35:29.822 INFO:journalctl@ceph.mgr.x.smithi003.stdout:-- Logs begin at Thu 2023-07-20 01:20:47 UTC. -- 2023-07-20T01:35:29.961 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:35:29.961 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537', '-v', 'bootstrap', '--fsid', '9b4fd2fc-269d-11ee-9b34-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.3', '--skip-admin-label'] 2023-07-20T01:35:30.105 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.1.1 2023-07-20T01:35:30.105 INFO:teuthology.orchestra.run.smithi003.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-20T01:35:30.106 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying podman|docker is present... 2023-07-20T01:35:30.247 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.1.1 2023-07-20T01:35:30.247 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying lvm2 is present... 2023-07-20T01:35:30.247 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying time synchronization is in place... 2023-07-20T01:35:30.254 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T01:35:30.254 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T01:35:30.260 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T01:35:30.260 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-07-20T01:35:30.266 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2023-07-20T01:35:30.272 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2023-07-20T01:35:30.272 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2023-07-20T01:35:30.272 INFO:teuthology.orchestra.run.smithi003.stdout:Repeating the final host check... 2023-07-20T01:35:30.418 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.1.1 2023-07-20T01:35:30.418 INFO:teuthology.orchestra.run.smithi003.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-20T01:35:30.418 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl is present 2023-07-20T01:35:30.418 INFO:teuthology.orchestra.run.smithi003.stdout:lvcreate is present 2023-07-20T01:35:30.424 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T01:35:30.424 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T01:35:30.430 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T01:35:30.430 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-07-20T01:35:30.436 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2023-07-20T01:35:30.441 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2023-07-20T01:35:30.442 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2023-07-20T01:35:30.442 INFO:teuthology.orchestra.run.smithi003.stdout:Host looks OK 2023-07-20T01:35:30.442 INFO:teuthology.orchestra.run.smithi003.stdout:Cluster fsid: 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:35:30.442 INFO:teuthology.orchestra.run.smithi003.stdout:Acquiring lock 140588033681896 on /run/cephadm/9b4fd2fc-269d-11ee-9b34-001a4aab830c.lock 2023-07-20T01:35:30.443 INFO:teuthology.orchestra.run.smithi003.stdout:Lock 140588033681896 acquired on /run/cephadm/9b4fd2fc-269d-11ee-9b34-001a4aab830c.lock 2023-07-20T01:35:30.443 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 3300 ... 2023-07-20T01:35:30.443 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 6789 ... 2023-07-20T01:35:30.443 INFO:teuthology.orchestra.run.smithi003.stdout:Base mon IP(s) is [172.21.15.3:3300, 172.21.15.3:6789], mon addrv is [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2023-07-20T01:35:30.446 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-07-20T01:35:30.447 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.3 metric 100 2023-07-20T01:35:30.449 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-20T01:35:30.449 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-07-20T01:35:30.449 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1775sec hoplimit 64 pref medium 2023-07-20T01:35:30.452 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-20T01:35:30.452 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-20T01:35:30.452 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T01:35:30.452 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-07-20T01:35:30.452 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15dc/64 scope link 2023-07-20T01:35:30.452 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T01:35:30.453 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2023-07-20T01:35:30.453 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2023-07-20T01:35:30.453 INFO:teuthology.orchestra.run.smithi003.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-20T01:35:30.454 INFO:teuthology.orchestra.run.smithi003.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-20T01:35:30.454 INFO:teuthology.orchestra.run.smithi003.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T01:35:30.998 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T01:35:30.999 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T01:35:30.999 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Getting image source signatures 2023-07-20T01:35:30.999 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:9a3af2d1f966fade87cf08b715899d7eff2dde16dabc6e2e4c3ec2059da8adc1 2023-07-20T01:35:30.999 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-20T01:35:31.000 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying config sha256:3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T01:35:31.000 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-20T01:35:31.000 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Storing signatures 2023-07-20T01:35:31.849 INFO:teuthology.orchestra.run.smithi003.stdout:ceph: stdout ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T01:35:31.849 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph version: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T01:35:31.849 INFO:teuthology.orchestra.run.smithi003.stdout:Extracting ceph user uid/gid from container image... 2023-07-20T01:35:32.597 INFO:teuthology.orchestra.run.smithi003.stdout:stat: stdout 167 167 2023-07-20T01:35:32.597 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial keys... 2023-07-20T01:35:33.349 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQBlj7hkDq8uABAA79FjDskYwfh9WC8UnCJyRA== 2023-07-20T01:35:34.099 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQBlj7hkhuwULBAAMClOypgC5IfFz8DjXn4fNw== 2023-07-20T01:35:34.848 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQBmj7hkxcknHhAAzC/P42ImRJnn99dRy2lN7A== 2023-07-20T01:35:34.849 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial monmap... 2023-07-20T01:35:35.595 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T01:35:35.595 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-20T01:35:35.596 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:35:35.596 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T01:35:35.596 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool for a [v2:172.21.15.3:3300,v1:172.21.15.3:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T01:35:35.596 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = pacific 2023-07-20T01:35:35.596 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: set fsid to 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:35:35.596 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T01:35:35.597 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:35:35.597 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mon... 2023-07-20T01:35:36.445 INFO:teuthology.orchestra.run.smithi003.stdout:create mon.a on 2023-07-20T01:35:36.581 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-20T01:35:36.676 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-20T01:35:36.798 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c.target. 2023-07-20T01:35:36.798 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c.target. 2023-07-20T01:35:37.105 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.a 2023-07-20T01:35:37.105 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.a.service: Unit ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.a.service not loaded. 2023-07-20T01:35:37.196 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c.target.wants/ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@.service. 2023-07-20T01:35:37.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:37 smithi003 systemd[1]: Starting Ceph mon.a for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:35:37.704 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T01:35:37.704 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2023-07-20T01:35:37.711 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T01:35:37.712 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-07-20T01:35:37.712 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2023-07-20T01:35:37.712 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T01:35:37.712 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon to start... 2023-07-20T01:35:37.713 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon... 2023-07-20T01:35:37.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:37 smithi003 podman[95886]: 2023-07-20T01:35:37.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:37 smithi003 bash[95886]: 25bee30a475f76fd3f0489b449e312b44fdbad3d0142d86c544795b158527a44 2023-07-20T01:35:37.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:37 smithi003 systemd[1]: Started Ceph mon.a for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:35:37.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:37 smithi003 ceph-mon[95926]: mkfs 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:35:37.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:37 smithi003 ceph-mon[95926]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T01:35:38.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout cluster: 2023-07-20T01:35:38.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout id: 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:35:38.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-20T01:35:38.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:35:38.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout services: 2023-07-20T01:35:38.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.672495s) 2023-07-20T01:35:38.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-20T01:35:38.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-20T01:35:38.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:35:38.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout data: 2023-07-20T01:35:38.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-20T01:35:38.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-20T01:35:38.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-20T01:35:38.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pgs: 2023-07-20T01:35:38.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:35:38.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.265+0000 7f7e05ee0700 1 Processor -- start 2023-07-20T01:35:38.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.265+0000 7f7e05ee0700 1 -- start start 2023-07-20T01:35:38.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.265+0000 7f7e05ee0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e00108e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.265+0000 7f7e05ee0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e00079e10 con 0x7f7e00106a80 2023-07-20T01:35:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.266+0000 7f7dff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e00108e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.266+0000 7f7dff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e00108e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41436/0 (socket says 172.21.15.3:41436) 2023-07-20T01:35:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.266+0000 7f7dff7fe700 1 -- 172.21.15.3:0/2546784130 learned_addr learned my addr 172.21.15.3:0/2546784130 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.266+0000 7f7dff7fe700 1 -- 172.21.15.3:0/2546784130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e001093b0 con 0x7f7e00106a80 2023-07-20T01:35:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.266+0000 7f7dff7fe700 1 --2- 172.21.15.3:0/2546784130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e00108e70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7df00098a0 tx=0x7f7df0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827ee24ba67c5db9 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.266+0000 7f7dfe7fc700 1 -- 172.21.15.3:0/2546784130 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7df0004160 con 0x7f7e00106a80 2023-07-20T01:35:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.266+0000 7f7dfe7fc700 1 -- 172.21.15.3:0/2546784130 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df00042c0 con 0x7f7e00106a80 2023-07-20T01:35:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.266+0000 7f7dfe7fc700 1 -- 172.21.15.3:0/2546784130 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7df00045a0 con 0x7f7e00106a80 2023-07-20T01:35:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.267+0000 7f7e05ee0700 1 -- 172.21.15.3:0/2546784130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 msgr2=0x7f7e00108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.267+0000 7f7e05ee0700 1 --2- 172.21.15.3:0/2546784130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e00108e70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7df00098a0 tx=0x7f7df0009bb0 comp rx=0 tx=0).stop 2023-07-20T01:35:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.267+0000 7f7e05ee0700 1 -- 172.21.15.3:0/2546784130 shutdown_connections 2023-07-20T01:35:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.267+0000 7f7e05ee0700 1 --2- 172.21.15.3:0/2546784130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e00108e70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:38.773 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.267+0000 7f7e05ee0700 1 -- 172.21.15.3:0/2546784130 >> 172.21.15.3:0/2546784130 conn(0x7f7e001004d0 msgr2=0x7f7e001028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:38.773 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.267+0000 7f7e05ee0700 1 -- 172.21.15.3:0/2546784130 shutdown_connections 2023-07-20T01:35:38.773 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.267+0000 7f7e05ee0700 1 -- 172.21.15.3:0/2546784130 wait complete. 2023-07-20T01:35:38.773 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.268+0000 7f7e05ee0700 1 Processor -- start 2023-07-20T01:35:38.773 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.268+0000 7f7e05ee0700 1 -- start start 2023-07-20T01:35:38.773 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.268+0000 7f7e05ee0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e0019b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:38.773 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.268+0000 7f7e05ee0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0019bc50 con 0x7f7e00106a80 2023-07-20T01:35:38.773 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.268+0000 7f7dff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e0019b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:38.774 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.268+0000 7f7dff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e0019b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41438/0 (socket says 172.21.15.3:41438) 2023-07-20T01:35:38.774 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.268+0000 7f7dff7fe700 1 -- 172.21.15.3:0/1762246850 learned_addr learned my addr 172.21.15.3:0/1762246850 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:38.774 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.269+0000 7f7dff7fe700 1 -- 172.21.15.3:0/1762246850 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df0009550 con 0x7f7e00106a80 2023-07-20T01:35:38.774 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.269+0000 7f7dff7fe700 1 --2- 172.21.15.3:0/1762246850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e0019b710 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7e00079250 tx=0x7f7df0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:38.774 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.269+0000 7f7dfcff9700 1 -- 172.21.15.3:0/1762246850 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7df0011630 con 0x7f7e00106a80 2023-07-20T01:35:38.774 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.269+0000 7f7dfcff9700 1 -- 172.21.15.3:0/1762246850 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df0011790 con 0x7f7e00106a80 2023-07-20T01:35:38.774 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.269+0000 7f7e05ee0700 1 -- 172.21.15.3:0/1762246850 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0019be50 con 0x7f7e00106a80 2023-07-20T01:35:38.775 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.269+0000 7f7dfcff9700 1 -- 172.21.15.3:0/1762246850 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7df0011a30 con 0x7f7e00106a80 2023-07-20T01:35:38.775 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.269+0000 7f7e05ee0700 1 -- 172.21.15.3:0/1762246850 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0019c270 con 0x7f7e00106a80 2023-07-20T01:35:38.775 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.269+0000 7f7dfcff9700 1 -- 172.21.15.3:0/1762246850 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7df0022070 con 0x7f7e00106a80 2023-07-20T01:35:38.775 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.269+0000 7f7dfcff9700 1 -- 172.21.15.3:0/1762246850 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7df001c450 con 0x7f7e00106a80 2023-07-20T01:35:38.775 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.270+0000 7f7e05ee0700 1 -- 172.21.15.3:0/1762246850 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e00195710 con 0x7f7e00106a80 2023-07-20T01:35:38.775 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.272+0000 7f7dfcff9700 1 -- 172.21.15.3:0/1762246850 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f7df001b450 con 0x7f7e00106a80 2023-07-20T01:35:38.775 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.403+0000 7f7e05ee0700 1 -- 172.21.15.3:0/1762246850 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f7e0002c970 con 0x7f7e00106a80 2023-07-20T01:35:38.775 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.404+0000 7f7dfcff9700 1 -- 172.21.15.3:0/1762246850 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f7df001b5f0 con 0x7f7e00106a80 2023-07-20T01:35:38.776 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.404+0000 7f7e05ee0700 1 -- 172.21.15.3:0/1762246850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 msgr2=0x7f7e0019b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:38.776 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.404+0000 7f7e05ee0700 1 --2- 172.21.15.3:0/1762246850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e0019b710 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7e00079250 tx=0x7f7df0004750 comp rx=0 tx=0).stop 2023-07-20T01:35:38.776 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.404+0000 7f7e05ee0700 1 -- 172.21.15.3:0/1762246850 shutdown_connections 2023-07-20T01:35:38.776 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.404+0000 7f7e05ee0700 1 --2- 172.21.15.3:0/1762246850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e00106a80 0x7f7e0019b710 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:38.776 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.404+0000 7f7e05ee0700 1 -- 172.21.15.3:0/1762246850 >> 172.21.15.3:0/1762246850 conn(0x7f7e001004d0 msgr2=0x7f7e00101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:38.776 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.405+0000 7f7e05ee0700 1 -- 172.21.15.3:0/1762246850 shutdown_connections 2023-07-20T01:35:38.777 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:38.405+0000 7f7e05ee0700 1 -- 172.21.15.3:0/1762246850 wait complete. 2023-07-20T01:35:38.777 INFO:teuthology.orchestra.run.smithi003.stdout:mon is available 2023-07-20T01:35:38.777 INFO:teuthology.orchestra.run.smithi003.stdout:Assimilating anything we can from ceph.conf... 2023-07-20T01:35:38.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:38 smithi003 ceph-mon[95926]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T01:35:38.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:38 smithi003 ceph-mon[95926]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} 2023-07-20T01:35:38.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:38 smithi003 ceph-mon[95926]: fsmap 2023-07-20T01:35:38.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:38 smithi003 ceph-mon[95926]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T01:35:38.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:38 smithi003 ceph-mon[95926]: mgrmap e1: no daemons active 2023-07-20T01:35:38.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:38 smithi003 ceph-mon[95926]: from='client.? 172.21.15.3:0/1762246850' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-20T01:35:39.706 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:35:39.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T01:35:39.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:35:39.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2023-07-20T01:35:39.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T01:35:39.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T01:35:39.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T01:35:39.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T01:35:39.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:35:39.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T01:35:39.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T01:35:39.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:35:39.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T01:35:39.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T01:35:39.713 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T01:35:39.713 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T01:35:39.713 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T01:35:39.714 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T01:35:39.714 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.276+0000 7f6ad1a34700 1 Processor -- start 2023-07-20T01:35:39.714 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.276+0000 7f6ad1a34700 1 -- start start 2023-07-20T01:35:39.714 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.276+0000 7f6ad1a34700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc108e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:39.714 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.276+0000 7f6ad1a34700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc073f70 con 0x7f6acc106a80 2023-07-20T01:35:39.714 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.277+0000 7f6acaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc108e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:39.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.277+0000 7f6acaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc108e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41440/0 (socket says 172.21.15.3:41440) 2023-07-20T01:35:39.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.277+0000 7f6acaffd700 1 -- 172.21.15.3:0/3854422612 learned_addr learned my addr 172.21.15.3:0/3854422612 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:39.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.277+0000 7f6acaffd700 1 -- 172.21.15.3:0/3854422612 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc1093b0 con 0x7f6acc106a80 2023-07-20T01:35:39.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.277+0000 7f6acaffd700 1 --2- 172.21.15.3:0/3854422612 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc108e70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00098a0 tx=0x7f6ac0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98914a9cf330506c server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:39.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.277+0000 7f6ac9ffb700 1 -- 172.21.15.3:0/3854422612 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ac0004160 con 0x7f6acc106a80 2023-07-20T01:35:39.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.278+0000 7f6ac9ffb700 1 -- 172.21.15.3:0/3854422612 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac00042c0 con 0x7f6acc106a80 2023-07-20T01:35:39.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.278+0000 7f6ac9ffb700 1 -- 172.21.15.3:0/3854422612 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ac00045a0 con 0x7f6acc106a80 2023-07-20T01:35:39.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.278+0000 7f6ad1a34700 1 -- 172.21.15.3:0/3854422612 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 msgr2=0x7f6acc108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:39.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.278+0000 7f6ad1a34700 1 --2- 172.21.15.3:0/3854422612 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc108e70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00098a0 tx=0x7f6ac0009bb0 comp rx=0 tx=0).stop 2023-07-20T01:35:39.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.278+0000 7f6ad1a34700 1 -- 172.21.15.3:0/3854422612 shutdown_connections 2023-07-20T01:35:39.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.278+0000 7f6ad1a34700 1 --2- 172.21.15.3:0/3854422612 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc108e70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:39.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.278+0000 7f6ad1a34700 1 -- 172.21.15.3:0/3854422612 >> 172.21.15.3:0/3854422612 conn(0x7f6acc100490 msgr2=0x7f6acc1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:39.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.278+0000 7f6ad1a34700 1 -- 172.21.15.3:0/3854422612 shutdown_connections 2023-07-20T01:35:39.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.278+0000 7f6ad1a34700 1 -- 172.21.15.3:0/3854422612 wait complete. 2023-07-20T01:35:39.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.279+0000 7f6ad1a34700 1 Processor -- start 2023-07-20T01:35:39.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.279+0000 7f6ad1a34700 1 -- start start 2023-07-20T01:35:39.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.279+0000 7f6ad1a34700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc1063e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:39.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.280+0000 7f6ad1a34700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc106920 con 0x7f6acc106a80 2023-07-20T01:35:39.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.280+0000 7f6acaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc1063e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:39.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.280+0000 7f6acaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc1063e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41442/0 (socket says 172.21.15.3:41442) 2023-07-20T01:35:39.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.280+0000 7f6acaffd700 1 -- 172.21.15.3:0/4177804398 learned_addr learned my addr 172.21.15.3:0/4177804398 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:39.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.280+0000 7f6acaffd700 1 -- 172.21.15.3:0/4177804398 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac0009550 con 0x7f6acc106a80 2023-07-20T01:35:39.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.280+0000 7f6acaffd700 1 --2- 172.21.15.3:0/4177804398 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc1063e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0733b0 tx=0x7f6ac0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:39.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.280+0000 7f6ad0a32700 1 -- 172.21.15.3:0/4177804398 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ac0011630 con 0x7f6acc106a80 2023-07-20T01:35:39.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.280+0000 7f6ad0a32700 1 -- 172.21.15.3:0/4177804398 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac0011790 con 0x7f6acc106a80 2023-07-20T01:35:39.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.280+0000 7f6ad1a34700 1 -- 172.21.15.3:0/4177804398 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc104a30 con 0x7f6acc106a80 2023-07-20T01:35:39.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.280+0000 7f6ad0a32700 1 -- 172.21.15.3:0/4177804398 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ac0011a30 con 0x7f6acc106a80 2023-07-20T01:35:39.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.280+0000 7f6ad1a34700 1 -- 172.21.15.3:0/4177804398 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc104e50 con 0x7f6acc106a80 2023-07-20T01:35:39.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.281+0000 7f6ad0a32700 1 -- 172.21.15.3:0/4177804398 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6ac0022070 con 0x7f6acc106a80 2023-07-20T01:35:39.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.281+0000 7f6ad0a32700 1 -- 172.21.15.3:0/4177804398 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6ac001c450 con 0x7f6acc106a80 2023-07-20T01:35:39.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.281+0000 7f6ad1a34700 1 -- 172.21.15.3:0/4177804398 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6acc105860 con 0x7f6acc106a80 2023-07-20T01:35:39.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.284+0000 7f6ad0a32700 1 -- 172.21.15.3:0/4177804398 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f6ac001b450 con 0x7f6acc106a80 2023-07-20T01:35:39.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.410+0000 7f6ad1a34700 1 -- 172.21.15.3:0/4177804398 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6acc1932e0 con 0x7f6acc106a80 2023-07-20T01:35:39.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.413+0000 7f6ad0a32700 1 -- 172.21.15.3:0/4177804398 <== mon.0 v2:172.21.15.3:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ac001b5f0 con 0x7f6acc106a80 2023-07-20T01:35:39.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.414+0000 7f6ad0a32700 1 -- 172.21.15.3:0/4177804398 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+427 (secure 0 0 0) 0x7f6acc1932e0 con 0x7f6acc106a80 2023-07-20T01:35:39.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.415+0000 7f6ad1a34700 1 -- 172.21.15.3:0/4177804398 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 msgr2=0x7f6acc1063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:39.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.415+0000 7f6ad1a34700 1 --2- 172.21.15.3:0/4177804398 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc1063e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0733b0 tx=0x7f6ac0004750 comp rx=0 tx=0).stop 2023-07-20T01:35:39.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.415+0000 7f6ad1a34700 1 -- 172.21.15.3:0/4177804398 shutdown_connections 2023-07-20T01:35:39.720 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.415+0000 7f6ad1a34700 1 --2- 172.21.15.3:0/4177804398 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6acc106a80 0x7f6acc1063e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:39.720 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.415+0000 7f6ad1a34700 1 -- 172.21.15.3:0/4177804398 >> 172.21.15.3:0/4177804398 conn(0x7f6acc100490 msgr2=0x7f6acc101e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:39.720 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.415+0000 7f6ad1a34700 1 -- 172.21.15.3:0/4177804398 shutdown_connections 2023-07-20T01:35:39.720 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:39.415+0000 7f6ad1a34700 1 -- 172.21.15.3:0/4177804398 wait complete. 2023-07-20T01:35:39.720 INFO:teuthology.orchestra.run.smithi003.stdout:Generating new minimal ceph.conf... 2023-07-20T01:35:39.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:39 smithi003 ceph-mon[95926]: from='client.? 172.21.15.3:0/4177804398' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T01:35:39.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:39 smithi003 ceph-mon[95926]: from='client.? 172.21.15.3:0/4177804398' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-20T01:35:40.755 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.234+0000 7f422b2a7700 1 Processor -- start 2023-07-20T01:35:40.755 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.234+0000 7f422b2a7700 1 -- start start 2023-07-20T01:35:40.756 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.235+0000 7f422b2a7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f4224104cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:40.756 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.235+0000 7f422b2a7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4224073f30 con 0x7f42241048b0 2023-07-20T01:35:40.757 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.235+0000 7f4229043700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f4224104cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:40.757 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.235+0000 7f4229043700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f4224104cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41444/0 (socket says 172.21.15.3:41444) 2023-07-20T01:35:40.757 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.235+0000 7f4229043700 1 -- 172.21.15.3:0/1455091210 learned_addr learned my addr 172.21.15.3:0/1455091210 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:40.757 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.235+0000 7f4229043700 1 -- 172.21.15.3:0/1455091210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4224105210 con 0x7f42241048b0 2023-07-20T01:35:40.758 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.235+0000 7f4229043700 1 --2- 172.21.15.3:0/1455091210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f4224104cd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f42200098a0 tx=0x7f4220009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f528f0d6d2eaee server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:40.758 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.235+0000 7f4217fff700 1 -- 172.21.15.3:0/1455091210 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4220004160 con 0x7f42241048b0 2023-07-20T01:35:40.758 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.235+0000 7f4217fff700 1 -- 172.21.15.3:0/1455091210 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42200042c0 con 0x7f42241048b0 2023-07-20T01:35:40.758 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.235+0000 7f4217fff700 1 -- 172.21.15.3:0/1455091210 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4220011a00 con 0x7f42241048b0 2023-07-20T01:35:40.759 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.236+0000 7f422b2a7700 1 -- 172.21.15.3:0/1455091210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 msgr2=0x7f4224104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:40.759 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.236+0000 7f422b2a7700 1 --2- 172.21.15.3:0/1455091210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f4224104cd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f42200098a0 tx=0x7f4220009bb0 comp rx=0 tx=0).stop 2023-07-20T01:35:40.759 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.236+0000 7f422b2a7700 1 -- 172.21.15.3:0/1455091210 shutdown_connections 2023-07-20T01:35:40.760 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.236+0000 7f422b2a7700 1 --2- 172.21.15.3:0/1455091210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f4224104cd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:40.760 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.236+0000 7f422b2a7700 1 -- 172.21.15.3:0/1455091210 >> 172.21.15.3:0/1455091210 conn(0x7f4224100490 msgr2=0x7f42241028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:40.760 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.236+0000 7f422b2a7700 1 -- 172.21.15.3:0/1455091210 shutdown_connections 2023-07-20T01:35:40.760 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.236+0000 7f422b2a7700 1 -- 172.21.15.3:0/1455091210 wait complete. 2023-07-20T01:35:40.761 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.237+0000 7f422b2a7700 1 Processor -- start 2023-07-20T01:35:40.761 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.237+0000 7f422b2a7700 1 -- start start 2023-07-20T01:35:40.761 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.237+0000 7f422b2a7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f422419bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.237+0000 7f422b2a7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422419c410 con 0x7f42241048b0 2023-07-20T01:35:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.237+0000 7f4229043700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f422419bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.237+0000 7f4229043700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f422419bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41446/0 (socket says 172.21.15.3:41446) 2023-07-20T01:35:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.237+0000 7f4229043700 1 -- 172.21.15.3:0/2145674464 learned_addr learned my addr 172.21.15.3:0/2145674464 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.237+0000 7f4229043700 1 -- 172.21.15.3:0/2145674464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4220009550 con 0x7f42241048b0 2023-07-20T01:35:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.238+0000 7f4229043700 1 --2- 172.21.15.3:0/2145674464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f422419bed0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4220006af0 tx=0x7f4220011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.238+0000 7f42167fc700 1 -- 172.21.15.3:0/2145674464 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4220003a40 con 0x7f42241048b0 2023-07-20T01:35:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.238+0000 7f42167fc700 1 -- 172.21.15.3:0/2145674464 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4220003ba0 con 0x7f42241048b0 2023-07-20T01:35:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.238+0000 7f422b2a7700 1 -- 172.21.15.3:0/2145674464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422419c610 con 0x7f42241048b0 2023-07-20T01:35:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.238+0000 7f42167fc700 1 -- 172.21.15.3:0/2145674464 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f422001a6e0 con 0x7f42241048b0 2023-07-20T01:35:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.238+0000 7f422b2a7700 1 -- 172.21.15.3:0/2145674464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422419ca30 con 0x7f42241048b0 2023-07-20T01:35:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.238+0000 7f42167fc700 1 -- 172.21.15.3:0/2145674464 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f422001ac50 con 0x7f42241048b0 2023-07-20T01:35:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.238+0000 7f42167fc700 1 -- 172.21.15.3:0/2145674464 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4220019700 con 0x7f42241048b0 2023-07-20T01:35:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.239+0000 7f422b2a7700 1 -- 172.21.15.3:0/2145674464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42241957a0 con 0x7f42241048b0 2023-07-20T01:35:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.241+0000 7f42167fc700 1 -- 172.21.15.3:0/2145674464 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f422002de30 con 0x7f42241048b0 2023-07-20T01:35:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.367+0000 7f422b2a7700 1 -- 172.21.15.3:0/2145674464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f422402c970 con 0x7f42241048b0 2023-07-20T01:35:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.368+0000 7f42167fc700 1 -- 172.21.15.3:0/2145674464 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7f4220024a60 con 0x7f42241048b0 2023-07-20T01:35:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.369+0000 7f422b2a7700 1 -- 172.21.15.3:0/2145674464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 msgr2=0x7f422419bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.369+0000 7f422b2a7700 1 --2- 172.21.15.3:0/2145674464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f422419bed0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4220006af0 tx=0x7f4220011ad0 comp rx=0 tx=0).stop 2023-07-20T01:35:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.369+0000 7f422b2a7700 1 -- 172.21.15.3:0/2145674464 shutdown_connections 2023-07-20T01:35:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.369+0000 7f422b2a7700 1 --2- 172.21.15.3:0/2145674464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42241048b0 0x7f422419bed0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.369+0000 7f422b2a7700 1 -- 172.21.15.3:0/2145674464 >> 172.21.15.3:0/2145674464 conn(0x7f4224100490 msgr2=0x7f4224190470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.369+0000 7f422b2a7700 1 -- 172.21.15.3:0/2145674464 shutdown_connections 2023-07-20T01:35:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:40.369+0000 7f422b2a7700 1 -- 172.21.15.3:0/2145674464 wait complete. 2023-07-20T01:35:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:Restarting the monitor... 2023-07-20T01:35:41.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:40 smithi003 ceph-mon[95926]: from='client.? 172.21.15.3:0/2145674464' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:35:41.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:40 smithi003 systemd[1]: Stopping Ceph mon.a for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:35:41.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:41 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-a[95902]: 2023-07-20T01:35:41.144+0000 7f73ce6a7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:35:41.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:41 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-a[95902]: 2023-07-20T01:35:41.144+0000 7f73ce6a7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-20T01:35:41.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:41 smithi003 bash[96255]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-a 2023-07-20T01:35:42.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:41 smithi003 systemd[1]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.a.service: Succeeded. 2023-07-20T01:35:42.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:41 smithi003 systemd[1]: Stopped Ceph mon.a for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:35:42.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:41 smithi003 systemd[1]: Starting Ceph mon.a for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:35:42.283 INFO:teuthology.orchestra.run.smithi003.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-20T01:35:42.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 podman[96382]: 2023-07-20T01:35:42.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 bash[96382]: e9d70ced4521db0e64b85ad2fe0160ccd9a6b93ad53f7d3069168deafb3a2fc4 2023-07-20T01:35:42.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T01:35:42.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T01:35:42.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: pidfile_write: ignore empty --pid-file 2023-07-20T01:35:42.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: load: jerasure load: lrc 2023-07-20T01:35:42.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: RocksDB version: 7.9.2 2023-07-20T01:35:42.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Git sha 0 2023-07-20T01:35:42.350 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T01:35:42.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: DB SUMMARY 2023-07-20T01:35:42.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: DB Session ID: BB2YNMFPJL6I0G4BPRWN 2023-07-20T01:35:42.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: CURRENT file: CURRENT 2023-07-20T01:35:42.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T01:35:42.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-20T01:35:42.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-20T01:35:42.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99023 ; 2023-07-20T01:35:42.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.error_if_exists: 0 2023-07-20T01:35:42.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.create_if_missing: 0 2023-07-20T01:35:42.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.paranoid_checks: 1 2023-07-20T01:35:42.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T01:35:42.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T01:35:42.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T01:35:42.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.env: 0x55e481d80c60 2023-07-20T01:35:42.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T01:35:42.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.info_log: 0x55e4832812c0 2023-07-20T01:35:42.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T01:35:42.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.statistics: (nil) 2023-07-20T01:35:42.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.use_fsync: 0 2023-07-20T01:35:42.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_log_file_size: 0 2023-07-20T01:35:42.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T01:35:42.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T01:35:42.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T01:35:42.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T01:35:42.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.allow_fallocate: 1 2023-07-20T01:35:42.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T01:35:42.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T01:35:42.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.use_direct_reads: 0 2023-07-20T01:35:42.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T01:35:42.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T01:35:42.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.db_log_dir: 2023-07-20T01:35:42.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.wal_dir: 2023-07-20T01:35:42.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T01:35:42.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T01:35:42.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T01:35:42.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T01:35:42.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T01:35:42.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T01:35:42.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T01:35:42.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T01:35:42.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.write_buffer_manager: 0x55e4831a23c0 2023-07-20T01:35:42.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T01:35:42.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T01:35:42.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T01:35:42.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T01:35:42.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T01:35:42.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T01:35:42.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T01:35:42.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T01:35:42.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.unordered_write: 0 2023-07-20T01:35:42.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T01:35:42.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T01:35:42.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T01:35:42.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T01:35:42.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.row_cache: None 2023-07-20T01:35:42.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.wal_filter: None 2023-07-20T01:35:42.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T01:35:42.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T01:35:42.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.two_write_queues: 0 2023-07-20T01:35:42.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T01:35:42.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.wal_compression: 0 2023-07-20T01:35:42.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.atomic_flush: 0 2023-07-20T01:35:42.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T01:35:42.359 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T01:35:42.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T01:35:42.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.log_readahead_size: 0 2023-07-20T01:35:42.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T01:35:42.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T01:35:42.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T01:35:42.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T01:35:42.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T01:35:42.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T01:35:42.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T01:35:42.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_background_jobs: 2 2023-07-20T01:35:42.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_background_compactions: -1 2023-07-20T01:35:42.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_subcompactions: 1 2023-07-20T01:35:42.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T01:35:42.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T01:35:42.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T01:35:42.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T01:35:42.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T01:35:42.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T01:35:42.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T01:35:42.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T01:35:42.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_open_files: -1 2023-07-20T01:35:42.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T01:35:42.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T01:35:42.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T01:35:42.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T01:35:42.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_background_flushes: -1 2023-07-20T01:35:42.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Compression algorithms supported: 2023-07-20T01:35:42.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T01:35:42.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: kZSTD supported: 0 2023-07-20T01:35:42.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: kXpressCompression supported: 0 2023-07-20T01:35:42.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T01:35:42.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: kZlibCompression supported: 1 2023-07-20T01:35:42.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: kSnappyCompression supported: 1 2023-07-20T01:35:42.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: kLZ4Compression supported: 1 2023-07-20T01:35:42.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: kBZip2Compression supported: 0 2023-07-20T01:35:42.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T01:35:42.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T01:35:42.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-20T01:35:42.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T01:35:42.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T01:35:42.368 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.merge_operator: 2023-07-20T01:35:42.368 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_filter: None 2023-07-20T01:35:42.368 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T01:35:42.368 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T01:35:42.368 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T01:35:42.368 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T01:35:42.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e4832813c0) 2023-07-20T01:35:42.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks: 1 2023-07-20T01:35:42.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T01:35:42.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T01:35:42.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_top_level_index_and_filter: 1 2023-07-20T01:35:42.369 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_type: 0 2023-07-20T01:35:42.370 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_index_type: 0 2023-07-20T01:35:42.370 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_shortening: 1 2023-07-20T01:35:42.370 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T01:35:42.370 INFO:journalctl@ceph.mon.a.smithi003.stdout: checksum: 4 2023-07-20T01:35:42.370 INFO:journalctl@ceph.mon.a.smithi003.stdout: no_block_cache: 0 2023-07-20T01:35:42.370 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache: 0x55e4832411f0 2023-07-20T01:35:42.370 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_name: BinnedLRUCache 2023-07-20T01:35:42.370 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_options: 2023-07-20T01:35:42.371 INFO:journalctl@ceph.mon.a.smithi003.stdout: capacity : 536870912 2023-07-20T01:35:42.371 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_shard_bits : 4 2023-07-20T01:35:42.371 INFO:journalctl@ceph.mon.a.smithi003.stdout: strict_capacity_limit : 0 2023-07-20T01:35:42.371 INFO:journalctl@ceph.mon.a.smithi003.stdout: high_pri_pool_ratio: 0.000 2023-07-20T01:35:42.371 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_compressed: (nil) 2023-07-20T01:35:42.371 INFO:journalctl@ceph.mon.a.smithi003.stdout: persistent_cache: (nil) 2023-07-20T01:35:42.371 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size: 4096 2023-07-20T01:35:42.372 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size_deviation: 10 2023-07-20T01:35:42.372 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_restart_interval: 16 2023-07-20T01:35:42.372 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_block_restart_interval: 1 2023-07-20T01:35:42.372 INFO:journalctl@ceph.mon.a.smithi003.stdout: metadata_block_size: 4096 2023-07-20T01:35:42.372 INFO:journalctl@ceph.mon.a.smithi003.stdout: partition_filters: 0 2023-07-20T01:35:42.372 INFO:journalctl@ceph.mon.a.smithi003.stdout: use_delta_encoding: 1 2023-07-20T01:35:42.372 INFO:journalctl@ceph.mon.a.smithi003.stdout: filter_policy: bloomfilter 2023-07-20T01:35:42.372 INFO:journalctl@ceph.mon.a.smithi003.stdout: whole_key_filtering: 1 2023-07-20T01:35:42.373 INFO:journalctl@ceph.mon.a.smithi003.stdout: verify_compression: 0 2023-07-20T01:35:42.373 INFO:journalctl@ceph.mon.a.smithi003.stdout: read_amp_bytes_per_bit: 0 2023-07-20T01:35:42.373 INFO:journalctl@ceph.mon.a.smithi003.stdout: format_version: 5 2023-07-20T01:35:42.373 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_index_compression: 1 2023-07-20T01:35:42.373 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_align: 0 2023-07-20T01:35:42.373 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_auto_readahead_size: 262144 2023-07-20T01:35:42.373 INFO:journalctl@ceph.mon.a.smithi003.stdout: prepopulate_block_cache: 0 2023-07-20T01:35:42.374 INFO:journalctl@ceph.mon.a.smithi003.stdout: initial_auto_readahead_size: 8192 2023-07-20T01:35:42.374 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T01:35:42.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T01:35:42.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T01:35:42.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compression: NoCompression 2023-07-20T01:35:42.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T01:35:42.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T01:35:42.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T01:35:42.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.num_levels: 7 2023-07-20T01:35:42.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T01:35:42.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T01:35:42.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T01:35:42.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T01:35:42.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T01:35:42.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T01:35:42.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T01:35:42.376 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T01:35:42.376 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T01:35:42.376 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T01:35:42.376 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T01:35:42.376 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T01:35:42.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T01:35:42.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T01:35:42.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T01:35:42.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T01:35:42.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T01:35:42.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T01:35:42.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T01:35:42.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T01:35:42.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T01:35:42.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T01:35:42.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T01:35:42.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T01:35:42.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T01:35:42.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T01:35:42.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T01:35:42.379 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T01:35:42.379 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T01:35:42.379 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T01:35:42.379 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T01:35:42.379 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T01:35:42.379 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T01:35:42.379 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T01:35:42.379 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T01:35:42.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T01:35:42.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T01:35:42.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T01:35:42.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T01:35:42.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T01:35:42.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T01:35:42.380 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T01:35:42.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T01:35:42.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T01:35:42.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T01:35:42.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T01:35:42.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T01:35:42.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T01:35:42.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.table_properties_collectors: 2023-07-20T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.inplace_update_support: 0 2023-07-20T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T01:35:42.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T01:35:42.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T01:35:42.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T01:35:42.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.bloom_locality: 0 2023-07-20T01:35:42.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.max_successive_merges: 0 2023-07-20T01:35:42.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T01:35:42.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T01:35:42.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T01:35:42.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T01:35:42.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.ttl: 2592000 2023-07-20T01:35:42.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T01:35:42.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T01:35:42.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T01:35:42.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.enable_blob_files: false 2023-07-20T01:35:42.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.min_blob_size: 0 2023-07-20T01:35:42.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T01:35:42.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T01:35:42.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T01:35:42.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T01:35:42.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T01:35:42.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T01:35:42.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T01:35:42.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T01:35:42.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-20T01:35:42.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-20T01:35:42.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b9b56622-d77a-4789-9b3a-9fb83f3bb4d4 2023-07-20T01:35:42.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689816942299877, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-20T01:35:42.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-20T01:35:42.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689816942300960, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94242, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 316, "table_properties": {"data_size": 92240, "index_size": 284, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13747, "raw_average_key_size": 49, "raw_value_size": 84737, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 275, "num_filter_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689816942, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b9b56622-d77a-4789-9b3a-9fb83f3bb4d4", "db_session_id": "BB2YNMFPJL6I0G4BPRWN", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-20T01:35:42.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689816942301032, "job": 1, "event": "recovery_finished"} 2023-07-20T01:35:42.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-20T01:35:42.387 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T01:35:42.387 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e4832da000 2023-07-20T01:35:42.387 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: DB pointer 0x55e4832c6000 2023-07-20T01:35:42.387 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T01:35:42.387 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T01:35:42.388 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** DB Stats ** 2023-07-20T01:35:42.388 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T01:35:42.388 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T01:35:42.388 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T01:35:42.388 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T01:35:42.388 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T01:35:42.388 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T01:35:42.389 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T01:35:42.389 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-07-20T01:35:42.389 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2023-07-20T01:35:42.389 INFO:journalctl@ceph.mon.a.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T01:35:42.389 INFO:journalctl@ceph.mon.a.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T01:35:42.389 INFO:journalctl@ceph.mon.a.smithi003.stdout: L0 2/0 93.87 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:35:42.389 INFO:journalctl@ceph.mon.a.smithi003.stdout: Sum 2/0 93.87 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:35:42.389 INFO:journalctl@ceph.mon.a.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:35:42.390 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-07-20T01:35:42.390 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2023-07-20T01:35:42.390 INFO:journalctl@ceph.mon.a.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T01:35:42.390 INFO:journalctl@ceph.mon.a.smithi003.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T01:35:42.390 INFO:journalctl@ceph.mon.a.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:35:42.390 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-07-20T01:35:42.390 INFO:journalctl@ceph.mon.a.smithi003.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T01:35:42.391 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-07-20T01:35:42.391 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T01:35:42.391 INFO:journalctl@ceph.mon.a.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T01:35:42.391 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T01:35:42.391 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T01:35:42.391 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T01:35:42.391 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T01:35:42.391 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative compaction: 0.00 GB write, 16.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T01:35:42.392 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval compaction: 0.00 GB write, 16.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T01:35:42.392 INFO:journalctl@ceph.mon.a.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T01:35:42.392 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache BinnedLRUCache@0x55e4832411f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-20T01:35:42.392 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-20T01:35:42.392 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-07-20T01:35:42.392 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T01:35:42.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: starting mon.a rank 0 at public addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] at bind addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:35:42.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: mon.a@-1(???) e1 preinit fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:35:42.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: mon.a@-1(???).mds e1 new map 2023-07-20T01:35:42.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: mon.a@-1(???).mds e1 print_map 2023-07-20T01:35:42.393 INFO:journalctl@ceph.mon.a.smithi003.stdout: e1 2023-07-20T01:35:42.393 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T01:35:42.393 INFO:journalctl@ceph.mon.a.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T01:35:42.393 INFO:journalctl@ceph.mon.a.smithi003.stdout: legacy client fscid: -1 2023-07-20T01:35:42.394 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-07-20T01:35:42.394 INFO:journalctl@ceph.mon.a.smithi003.stdout: No filesystems configured 2023-07-20T01:35:42.394 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T01:35:42.394 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:35:42.394 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:35:42.394 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:35:42.394 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-20T01:35:42.394 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta expand map: {default=false} 2023-07-20T01:35:42.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta from 'false' to 'false' 2023-07-20T01:35:42.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta expanded map: {default=false} 2023-07-20T01:35:42.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta expand map: {default=info} 2023-07-20T01:35:42.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta from 'info' to 'info' 2023-07-20T01:35:42.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta expanded map: {default=info} 2023-07-20T01:35:42.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta expand map: {default=daemon} 2023-07-20T01:35:42.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T01:35:42.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T01:35:42.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta expand map: {default=debug} 2023-07-20T01:35:42.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T01:35:42.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: expand_channel_meta expanded map: {default=debug} 2023-07-20T01:35:42.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 systemd[1]: Started Ceph mon.a for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:35:42.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T01:35:42.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} 2023-07-20T01:35:42.396 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: fsmap 2023-07-20T01:35:42.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T01:35:42.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:42 smithi003 ceph-mon[96422]: mgrmap e1: no daemons active 2023-07-20T01:35:43.334 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.857+0000 7ff05f34f700 1 Processor -- start 2023-07-20T01:35:43.334 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.857+0000 7ff05f34f700 1 -- start start 2023-07-20T01:35:43.335 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.857+0000 7ff05f34f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff058108e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:43.335 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.857+0000 7ff05f34f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058079e10 con 0x7ff058106a80 2023-07-20T01:35:43.335 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.857+0000 7ff05d0eb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff058108e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:43.335 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.858+0000 7ff05d0eb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff058108e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41448/0 (socket says 172.21.15.3:41448) 2023-07-20T01:35:43.335 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.858+0000 7ff05d0eb700 1 -- 172.21.15.3:0/2712962684 learned_addr learned my addr 172.21.15.3:0/2712962684 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:43.335 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.858+0000 7ff05d0eb700 1 -- 172.21.15.3:0/2712962684 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0581093b0 con 0x7ff058106a80 2023-07-20T01:35:43.335 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.858+0000 7ff05d0eb700 1 --2- 172.21.15.3:0/2712962684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff058108e70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff0540098a0 tx=0x7ff054009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6377df8318085324 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:43.336 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.858+0000 7ff04ffff700 1 -- 172.21.15.3:0/2712962684 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff054004160 con 0x7ff058106a80 2023-07-20T01:35:43.336 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.858+0000 7ff04ffff700 1 -- 172.21.15.3:0/2712962684 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0540042c0 con 0x7ff058106a80 2023-07-20T01:35:43.336 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.858+0000 7ff04ffff700 1 -- 172.21.15.3:0/2712962684 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff054011a00 con 0x7ff058106a80 2023-07-20T01:35:43.336 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.859+0000 7ff05f34f700 1 -- 172.21.15.3:0/2712962684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 msgr2=0x7ff058108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:43.336 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.859+0000 7ff05f34f700 1 --2- 172.21.15.3:0/2712962684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff058108e70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff0540098a0 tx=0x7ff054009bb0 comp rx=0 tx=0).stop 2023-07-20T01:35:43.336 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.859+0000 7ff05f34f700 1 -- 172.21.15.3:0/2712962684 shutdown_connections 2023-07-20T01:35:43.336 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.859+0000 7ff05f34f700 1 --2- 172.21.15.3:0/2712962684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff058108e70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:43.337 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.859+0000 7ff05f34f700 1 -- 172.21.15.3:0/2712962684 >> 172.21.15.3:0/2712962684 conn(0x7ff0581004d0 msgr2=0x7ff0581028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:43.337 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.859+0000 7ff05f34f700 1 -- 172.21.15.3:0/2712962684 shutdown_connections 2023-07-20T01:35:43.337 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.859+0000 7ff05f34f700 1 -- 172.21.15.3:0/2712962684 wait complete. 2023-07-20T01:35:43.337 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.860+0000 7ff05f34f700 1 Processor -- start 2023-07-20T01:35:43.337 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.860+0000 7ff05f34f700 1 -- start start 2023-07-20T01:35:43.337 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.860+0000 7ff05f34f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff0581063e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:43.337 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.860+0000 7ff05f34f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058106920 con 0x7ff058106a80 2023-07-20T01:35:43.337 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.860+0000 7ff05d0eb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff0581063e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:43.338 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.860+0000 7ff05d0eb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff0581063e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41450/0 (socket says 172.21.15.3:41450) 2023-07-20T01:35:43.338 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.860+0000 7ff05d0eb700 1 -- 172.21.15.3:0/2986868207 learned_addr learned my addr 172.21.15.3:0/2986868207 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:43.338 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.860+0000 7ff05d0eb700 1 -- 172.21.15.3:0/2986868207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff054009550 con 0x7ff058106a80 2023-07-20T01:35:43.338 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.860+0000 7ff05d0eb700 1 --2- 172.21.15.3:0/2986868207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff0581063e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff054009fb0 tx=0x7ff054011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:43.338 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.861+0000 7ff04e7fc700 1 -- 172.21.15.3:0/2986868207 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff054003a40 con 0x7ff058106a80 2023-07-20T01:35:43.338 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.861+0000 7ff04e7fc700 1 -- 172.21.15.3:0/2986868207 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff054003ba0 con 0x7ff058106a80 2023-07-20T01:35:43.338 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.861+0000 7ff04e7fc700 1 -- 172.21.15.3:0/2986868207 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff05401a750 con 0x7ff058106a80 2023-07-20T01:35:43.338 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.861+0000 7ff05f34f700 1 -- 172.21.15.3:0/2986868207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff058104a30 con 0x7ff058106a80 2023-07-20T01:35:43.339 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.861+0000 7ff05f34f700 1 -- 172.21.15.3:0/2986868207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff058104e50 con 0x7ff058106a80 2023-07-20T01:35:43.339 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.861+0000 7ff04e7fc700 1 -- 172.21.15.3:0/2986868207 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff05401acd0 con 0x7ff058106a80 2023-07-20T01:35:43.339 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.861+0000 7ff04e7fc700 1 -- 172.21.15.3:0/2986868207 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff0540193c0 con 0x7ff058106a80 2023-07-20T01:35:43.339 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.862+0000 7ff05f34f700 1 -- 172.21.15.3:0/2986868207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff058105860 con 0x7ff058106a80 2023-07-20T01:35:43.339 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.864+0000 7ff04e7fc700 1 -- 172.21.15.3:0/2986868207 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7ff054024b60 con 0x7ff058106a80 2023-07-20T01:35:43.339 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.989+0000 7ff05f34f700 1 -- 172.21.15.3:0/2986868207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7ff0581937d0 con 0x7ff058106a80 2023-07-20T01:35:43.339 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.992+0000 7ff04e7fc700 1 -- 172.21.15.3:0/2986868207 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff0581937d0 con 0x7ff058106a80 2023-07-20T01:35:43.340 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.992+0000 7ff05f34f700 1 -- 172.21.15.3:0/2986868207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 msgr2=0x7ff0581063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:43.340 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.992+0000 7ff05f34f700 1 --2- 172.21.15.3:0/2986868207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff0581063e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff054009fb0 tx=0x7ff054011ad0 comp rx=0 tx=0).stop 2023-07-20T01:35:43.340 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.992+0000 7ff05f34f700 1 -- 172.21.15.3:0/2986868207 shutdown_connections 2023-07-20T01:35:43.340 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.992+0000 7ff05f34f700 1 --2- 172.21.15.3:0/2986868207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106a80 0x7ff0581063e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:43.340 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.992+0000 7ff05f34f700 1 -- 172.21.15.3:0/2986868207 >> 172.21.15.3:0/2986868207 conn(0x7ff0581004d0 msgr2=0x7ff058101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:43.340 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.993+0000 7ff05f34f700 1 -- 172.21.15.3:0/2986868207 shutdown_connections 2023-07-20T01:35:43.340 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:42.993+0000 7ff05f34f700 1 -- 172.21.15.3:0/2986868207 wait complete. 2023-07-20T01:35:43.340 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-20T01:35:43.352 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:35:43.352 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mgr... 2023-07-20T01:35:43.352 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 9283 ... 2023-07-20T01:35:43.353 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 8765 ... 2023-07-20T01:35:43.535 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T01:35:43.535 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.x.service: Unit ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.x.service not loaded. 2023-07-20T01:35:43.629 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c.target.wants/ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@.service. 2023-07-20T01:35:43.930 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:43 smithi003 systemd[1]: Starting Ceph mgr.x for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:35:44.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:43 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2986868207' entity='client.admin' 2023-07-20T01:35:44.191 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:43 smithi003 podman[96657]: 2023-07-20T01:35:44.191 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:44 smithi003 bash[96657]: 037b44da0bdf770647df17ea3d1d4885a4658f4a01700e9a477dc33c3148ebcb 2023-07-20T01:35:44.237 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T01:35:44.238 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2023-07-20T01:35:44.245 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T01:35:44.245 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-07-20T01:35:44.245 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2023-07-20T01:35:44.245 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T01:35:44.253 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T01:35:44.254 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2023-07-20T01:35:44.261 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T01:35:44.261 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-07-20T01:35:44.261 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2023-07-20T01:35:44.262 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-20T01:35:44.262 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr to start... 2023-07-20T01:35:44.262 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr... 2023-07-20T01:35:44.484 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:44 smithi003 systemd[1]: Started Ceph mgr.x for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:35:44.484 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:44 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:44.351+0000 7f71655a8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T01:35:44.485 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:44 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:44.447+0000 7f71655a8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T01:35:44.856 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:44 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:44.560+0000 7f71655a8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T01:35:44.856 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:44 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:44.672+0000 7f71655a8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T01:35:44.856 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:44 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:44.884+0000 7f71655a8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T01:35:45.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:44 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/92488880' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T01:35:45.208 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:35:45.210 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:35:45.210 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "9b4fd2fc-269d-11ee-9b34-001a4aab830c", 2023-07-20T01:35:45.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T01:35:45.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T01:35:45.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T01:35:45.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T01:35:45.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:45.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T01:35:45.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T01:35:45.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2023-07-20T01:35:45.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:45.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T01:35:45.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T01:35:45.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:45.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-07-20T01:35:45.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T01:35:45.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:45.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T01:35:45.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T01:35:45.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:45.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T01:35:45.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:45.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T01:35:45.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T01:35:45.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T01:35:45.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T01:35:45.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T01:35:45.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T01:35:45.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:45.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T01:35:45.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T01:35:45.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T01:35:45.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T01:35:45.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T01:35:45.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T01:35:45.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T01:35:45.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T01:35:45.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T01:35:45.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:45.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T01:35:45.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:45.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T01:35:45.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T01:35:45.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:45.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T01:35:45.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T01:35:45.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T01:35:45.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T01:35:45.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T01:35:45.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T01:35:45.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T01:35:45.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:45.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:35:45.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:45.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T01:35:45.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:45.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T01:35:37.747335+0000", 2023-07-20T01:35:45.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:35:45.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:45.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T01:35:45.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:35:45.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.819+0000 7faef81c2700 1 Processor -- start 2023-07-20T01:35:45.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.819+0000 7faef81c2700 1 -- start start 2023-07-20T01:35:45.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.819+0000 7faef81c2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0108e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:45.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.819+0000 7faef81c2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef0073fb0 con 0x7faef0106a40 2023-07-20T01:35:45.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.820+0000 7faef5f5e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0108e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:45.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.820+0000 7faef5f5e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0108e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41456/0 (socket says 172.21.15.3:41456) 2023-07-20T01:35:45.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.820+0000 7faef5f5e700 1 -- 172.21.15.3:0/3803140573 learned_addr learned my addr 172.21.15.3:0/3803140573 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:45.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.820+0000 7faef5f5e700 1 -- 172.21.15.3:0/3803140573 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef0109370 con 0x7faef0106a40 2023-07-20T01:35:45.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.820+0000 7faef5f5e700 1 --2- 172.21.15.3:0/3803140573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0108e30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7faeec0098a0 tx=0x7faeec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e4758d4432f78 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:45.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.820+0000 7faef4f5c700 1 -- 172.21.15.3:0/3803140573 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faeec004160 con 0x7faef0106a40 2023-07-20T01:35:45.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.820+0000 7faef4f5c700 1 -- 172.21.15.3:0/3803140573 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faeec0042c0 con 0x7faef0106a40 2023-07-20T01:35:45.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.820+0000 7faef4f5c700 1 -- 172.21.15.3:0/3803140573 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faeec011a00 con 0x7faef0106a40 2023-07-20T01:35:45.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.821+0000 7faef81c2700 1 -- 172.21.15.3:0/3803140573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 msgr2=0x7faef0108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:45.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.821+0000 7faef81c2700 1 --2- 172.21.15.3:0/3803140573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0108e30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7faeec0098a0 tx=0x7faeec009bb0 comp rx=0 tx=0).stop 2023-07-20T01:35:45.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.821+0000 7faef81c2700 1 -- 172.21.15.3:0/3803140573 shutdown_connections 2023-07-20T01:35:45.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.821+0000 7faef81c2700 1 --2- 172.21.15.3:0/3803140573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0108e30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:45.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.821+0000 7faef81c2700 1 -- 172.21.15.3:0/3803140573 >> 172.21.15.3:0/3803140573 conn(0x7faef0100490 msgr2=0x7faef01028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:45.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.821+0000 7faef81c2700 1 -- 172.21.15.3:0/3803140573 shutdown_connections 2023-07-20T01:35:45.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.821+0000 7faef81c2700 1 -- 172.21.15.3:0/3803140573 wait complete. 2023-07-20T01:35:45.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.822+0000 7faef81c2700 1 Processor -- start 2023-07-20T01:35:45.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.822+0000 7faef81c2700 1 -- start start 2023-07-20T01:35:45.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.822+0000 7faef81c2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:45.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.822+0000 7faef81c2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef019a230 con 0x7faef0106a40 2023-07-20T01:35:45.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.822+0000 7faef5f5e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:45.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.822+0000 7faef5f5e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41458/0 (socket says 172.21.15.3:41458) 2023-07-20T01:35:45.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.822+0000 7faef5f5e700 1 -- 172.21.15.3:0/92488880 learned_addr learned my addr 172.21.15.3:0/92488880 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:45.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.823+0000 7faef5f5e700 1 -- 172.21.15.3:0/92488880 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec009550 con 0x7faef0106a40 2023-07-20T01:35:45.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.823+0000 7faef5f5e700 1 --2- 172.21.15.3:0/92488880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0199cf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faeec009fb0 tx=0x7faeec011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:45.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.823+0000 7faee6ffd700 1 -- 172.21.15.3:0/92488880 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faeec003930 con 0x7faef0106a40 2023-07-20T01:35:45.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.823+0000 7faee6ffd700 1 -- 172.21.15.3:0/92488880 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faeec003a90 con 0x7faef0106a40 2023-07-20T01:35:45.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.823+0000 7faef81c2700 1 -- 172.21.15.3:0/92488880 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef019a430 con 0x7faef0106a40 2023-07-20T01:35:45.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.823+0000 7faee6ffd700 1 -- 172.21.15.3:0/92488880 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faeec019610 con 0x7faef0106a40 2023-07-20T01:35:45.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.823+0000 7faef81c2700 1 -- 172.21.15.3:0/92488880 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef019a850 con 0x7faef0106a40 2023-07-20T01:35:45.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.824+0000 7faee6ffd700 1 -- 172.21.15.3:0/92488880 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7faeec0198c0 con 0x7faef0106a40 2023-07-20T01:35:45.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.824+0000 7faee6ffd700 1 -- 172.21.15.3:0/92488880 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7faeec02da60 con 0x7faef0106a40 2023-07-20T01:35:45.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.824+0000 7faef81c2700 1 -- 172.21.15.3:0/92488880 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef0061da0 con 0x7faef0106a40 2023-07-20T01:35:45.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.827+0000 7faee6ffd700 1 -- 172.21.15.3:0/92488880 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7faeec003c00 con 0x7faef0106a40 2023-07-20T01:35:45.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.959+0000 7faef81c2700 1 -- 172.21.15.3:0/92488880 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7faef019d3b0 con 0x7faef0106a40 2023-07-20T01:35:45.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.960+0000 7faee6ffd700 1 -- 172.21.15.3:0/92488880 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7faeec019d00 con 0x7faef0106a40 2023-07-20T01:35:45.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.961+0000 7faef81c2700 1 -- 172.21.15.3:0/92488880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 msgr2=0x7faef0199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:45.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.961+0000 7faef81c2700 1 --2- 172.21.15.3:0/92488880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0199cf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faeec009fb0 tx=0x7faeec011ad0 comp rx=0 tx=0).stop 2023-07-20T01:35:45.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.961+0000 7faef81c2700 1 -- 172.21.15.3:0/92488880 shutdown_connections 2023-07-20T01:35:45.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.961+0000 7faef81c2700 1 --2- 172.21.15.3:0/92488880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef0106a40 0x7faef0199cf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:45.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.961+0000 7faef81c2700 1 -- 172.21.15.3:0/92488880 >> 172.21.15.3:0/92488880 conn(0x7faef0100490 msgr2=0x7faef0101b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:45.230 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.961+0000 7faef81c2700 1 -- 172.21.15.3:0/92488880 shutdown_connections 2023-07-20T01:35:45.230 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:44.961+0000 7faef81c2700 1 -- 172.21.15.3:0/92488880 wait complete. 2023-07-20T01:35:45.230 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (1/15)... 2023-07-20T01:35:45.524 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:45 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:45.294+0000 7f71655a8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T01:35:45.524 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:45 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:45.402+0000 7f71655a8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T01:35:45.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:45 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:45.552+0000 7f71655a8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T01:35:45.941 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:45 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:45.667+0000 7f71655a8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T01:35:46.440 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:46 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:46.176+0000 7f71655a8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T01:35:47.190 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:46 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:46.870+0000 7f71655a8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T01:35:47.190 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:47 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:47.060+0000 7f71655a8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T01:35:48.191 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:48 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:48.051+0000 7f71655a8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:35:48.191 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:48 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:48.161+0000 7f71655a8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T01:35:48.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:35:48.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:35:48.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "9b4fd2fc-269d-11ee-9b34-001a4aab830c", 2023-07-20T01:35:48.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T01:35:48.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T01:35:48.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T01:35:48.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T01:35:48.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:48.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T01:35:48.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T01:35:48.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2023-07-20T01:35:48.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:48.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T01:35:48.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T01:35:48.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:48.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-07-20T01:35:48.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T01:35:48.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:48.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T01:35:48.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T01:35:48.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:48.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T01:35:48.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:48.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T01:35:48.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T01:35:48.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T01:35:48.269 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T01:35:48.269 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T01:35:48.269 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T01:35:48.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:48.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T01:35:48.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T01:35:48.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T01:35:48.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T01:35:48.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T01:35:48.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T01:35:48.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T01:35:48.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T01:35:48.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T01:35:48.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:48.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T01:35:48.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:48.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T01:35:48.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T01:35:48.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:48.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T01:35:48.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T01:35:48.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T01:35:48.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T01:35:48.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T01:35:48.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T01:35:48.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T01:35:48.273 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:48.273 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:35:48.273 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:48.273 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T01:35:48.273 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:48.273 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T01:35:37.747335+0000", 2023-07-20T01:35:48.273 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:35:48.273 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:48.274 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T01:35:48.274 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:35:48.274 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.749+0000 7f09726ef700 1 Processor -- start 2023-07-20T01:35:48.274 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.749+0000 7f09726ef700 1 -- start start 2023-07-20T01:35:48.274 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.749+0000 7f09726ef700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c1052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:48.274 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.749+0000 7f09726ef700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c1057e0 con 0x7f096c104e80 2023-07-20T01:35:48.274 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.749+0000 7f096bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c1052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:48.274 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.749+0000 7f096bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c1052a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41460/0 (socket says 172.21.15.3:41460) 2023-07-20T01:35:48.275 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.749+0000 7f096bfff700 1 -- 172.21.15.3:0/3553370295 learned_addr learned my addr 172.21.15.3:0/3553370295 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:48.275 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.749+0000 7f096bfff700 1 -- 172.21.15.3:0/3553370295 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c105920 con 0x7f096c104e80 2023-07-20T01:35:48.275 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.749+0000 7f096bfff700 1 --2- 172.21.15.3:0/3553370295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c1052a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f095c0098a0 tx=0x7f095c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f765b1f28fe0d3 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:48.275 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.750+0000 7f096affd700 1 -- 172.21.15.3:0/3553370295 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f095c004160 con 0x7f096c104e80 2023-07-20T01:35:48.275 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.750+0000 7f096affd700 1 -- 172.21.15.3:0/3553370295 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f095c0042c0 con 0x7f096c104e80 2023-07-20T01:35:48.275 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.750+0000 7f096affd700 1 -- 172.21.15.3:0/3553370295 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f095c011a00 con 0x7f096c104e80 2023-07-20T01:35:48.275 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.750+0000 7f09726ef700 1 -- 172.21.15.3:0/3553370295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 msgr2=0x7f096c1052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:48.276 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.750+0000 7f09726ef700 1 --2- 172.21.15.3:0/3553370295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c1052a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f095c0098a0 tx=0x7f095c009bb0 comp rx=0 tx=0).stop 2023-07-20T01:35:48.276 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.750+0000 7f09726ef700 1 -- 172.21.15.3:0/3553370295 shutdown_connections 2023-07-20T01:35:48.276 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.750+0000 7f09726ef700 1 --2- 172.21.15.3:0/3553370295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c1052a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:48.276 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.750+0000 7f09726ef700 1 -- 172.21.15.3:0/3553370295 >> 172.21.15.3:0/3553370295 conn(0x7f096c100490 msgr2=0x7f096c1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:48.276 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.750+0000 7f09726ef700 1 -- 172.21.15.3:0/3553370295 shutdown_connections 2023-07-20T01:35:48.276 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.751+0000 7f09726ef700 1 -- 172.21.15.3:0/3553370295 wait complete. 2023-07-20T01:35:48.276 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.751+0000 7f09726ef700 1 Processor -- start 2023-07-20T01:35:48.276 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.751+0000 7f09726ef700 1 -- start start 2023-07-20T01:35:48.277 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.752+0000 7f09726ef700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c073c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:48.277 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.752+0000 7f09726ef700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c19ca40 con 0x7f096c104e80 2023-07-20T01:35:48.277 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.752+0000 7f096bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c073c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:48.277 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.752+0000 7f096bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c073c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41462/0 (socket says 172.21.15.3:41462) 2023-07-20T01:35:48.277 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.752+0000 7f096bfff700 1 -- 172.21.15.3:0/114376614 learned_addr learned my addr 172.21.15.3:0/114376614 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:48.277 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.752+0000 7f096bfff700 1 -- 172.21.15.3:0/114376614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c009550 con 0x7f096c104e80 2023-07-20T01:35:48.277 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.752+0000 7f096bfff700 1 --2- 172.21.15.3:0/114376614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c073c30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f095c009fb0 tx=0x7f095c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:48.277 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.753+0000 7f09697fa700 1 -- 172.21.15.3:0/114376614 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f095c003930 con 0x7f096c104e80 2023-07-20T01:35:48.278 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.753+0000 7f09697fa700 1 -- 172.21.15.3:0/114376614 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f095c003a90 con 0x7f096c104e80 2023-07-20T01:35:48.278 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.753+0000 7f09726ef700 1 -- 172.21.15.3:0/114376614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096c19cb80 con 0x7f096c104e80 2023-07-20T01:35:48.278 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.753+0000 7f09697fa700 1 -- 172.21.15.3:0/114376614 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f095c0195e0 con 0x7f096c104e80 2023-07-20T01:35:48.278 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.753+0000 7f09726ef700 1 -- 172.21.15.3:0/114376614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096c19cfa0 con 0x7f096c104e80 2023-07-20T01:35:48.278 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.753+0000 7f09697fa700 1 -- 172.21.15.3:0/114376614 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f095c019b70 con 0x7f096c104e80 2023-07-20T01:35:48.278 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.753+0000 7f09697fa700 1 -- 172.21.15.3:0/114376614 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f095c0243f0 con 0x7f096c104e80 2023-07-20T01:35:48.278 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.754+0000 7f09726ef700 1 -- 172.21.15.3:0/114376614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096c061da0 con 0x7f096c104e80 2023-07-20T01:35:48.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.756+0000 7f09697fa700 1 -- 172.21.15.3:0/114376614 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f095c003c00 con 0x7f096c104e80 2023-07-20T01:35:48.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.889+0000 7f09726ef700 1 -- 172.21.15.3:0/114376614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f096c106cf0 con 0x7f096c104e80 2023-07-20T01:35:48.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.889+0000 7f09697fa700 1 -- 172.21.15.3:0/114376614 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f095c01a9d0 con 0x7f096c104e80 2023-07-20T01:35:48.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.890+0000 7f09726ef700 1 -- 172.21.15.3:0/114376614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 msgr2=0x7f096c073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:48.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.890+0000 7f09726ef700 1 --2- 172.21.15.3:0/114376614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c073c30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f095c009fb0 tx=0x7f095c011ad0 comp rx=0 tx=0).stop 2023-07-20T01:35:48.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.890+0000 7f09726ef700 1 -- 172.21.15.3:0/114376614 shutdown_connections 2023-07-20T01:35:48.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.890+0000 7f09726ef700 1 --2- 172.21.15.3:0/114376614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f096c104e80 0x7f096c073c30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:48.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.890+0000 7f09726ef700 1 -- 172.21.15.3:0/114376614 >> 172.21.15.3:0/114376614 conn(0x7f096c100490 msgr2=0x7f096c101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:48.280 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.890+0000 7f09726ef700 1 -- 172.21.15.3:0/114376614 shutdown_connections 2023-07-20T01:35:48.280 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:47.890+0000 7f09726ef700 1 -- 172.21.15.3:0/114376614 wait complete. 2023-07-20T01:35:48.280 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (2/15)... 2023-07-20T01:35:48.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:48 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/114376614' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T01:35:48.690 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:48 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:48.381+0000 7f71655a8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:35:48.953 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:48 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:48.728+0000 7f71655a8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T01:35:49.440 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:48 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:48.981+0000 7f71655a8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T01:35:49.440 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:49 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:49.162+0000 7f71655a8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T01:35:49.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:49 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:49.489+0000 7f71655a8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T01:35:50.534 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:50 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:50.310+0000 7f71655a8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T01:35:50.534 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:50 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:50.470+0000 7f71655a8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T01:35:50.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:50 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3183392991' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T01:35:50.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:50 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:50.718+0000 7f71655a8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T01:35:50.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:50 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:50.858+0000 7f71655a8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T01:35:51.210 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:35:51.210 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:35:51.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "9b4fd2fc-269d-11ee-9b34-001a4aab830c", 2023-07-20T01:35:51.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T01:35:51.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T01:35:51.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T01:35:51.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T01:35:51.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:51.211 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T01:35:51.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T01:35:51.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2023-07-20T01:35:51.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:51.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T01:35:51.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T01:35:51.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:51.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-07-20T01:35:51.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T01:35:51.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:51.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T01:35:51.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T01:35:51.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:51.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T01:35:51.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:51.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T01:35:51.214 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T01:35:51.214 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T01:35:51.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T01:35:51.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T01:35:51.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T01:35:51.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:51.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T01:35:51.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T01:35:51.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T01:35:51.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T01:35:51.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T01:35:51.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T01:35:51.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T01:35:51.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T01:35:51.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T01:35:51.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:51.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T01:35:51.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:51.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T01:35:51.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T01:35:51.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:51.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T01:35:51.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T01:35:51.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T01:35:51.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T01:35:51.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T01:35:51.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T01:35:51.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T01:35:51.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:51.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:35:51.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:51.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T01:35:51.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:51.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T01:35:37.747335+0000", 2023-07-20T01:35:51.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:35:51.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:51.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T01:35:51.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:35:51.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.774+0000 7f0d6017b700 1 Processor -- start 2023-07-20T01:35:51.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.774+0000 7f0d6017b700 1 -- start start 2023-07-20T01:35:51.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.775+0000 7f0d6017b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58106410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:51.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.775+0000 7f0d6017b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d581069e0 con 0x7f0d58105ff0 2023-07-20T01:35:51.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.775+0000 7f0d5df17700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58106410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:51.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.775+0000 7f0d5df17700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58106410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41464/0 (socket says 172.21.15.3:41464) 2023-07-20T01:35:51.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.775+0000 7f0d5df17700 1 -- 172.21.15.3:0/1014404088 learned_addr learned my addr 172.21.15.3:0/1014404088 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:51.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.775+0000 7f0d5df17700 1 -- 172.21.15.3:0/1014404088 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d581071f0 con 0x7f0d58105ff0 2023-07-20T01:35:51.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.775+0000 7f0d5df17700 1 --2- 172.21.15.3:0/1014404088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58106410 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0d540098a0 tx=0x7f0d54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63a3ccfd9f8c91 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:51.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.775+0000 7f0d5cf15700 1 -- 172.21.15.3:0/1014404088 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d54004160 con 0x7f0d58105ff0 2023-07-20T01:35:51.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.776+0000 7f0d5cf15700 1 -- 172.21.15.3:0/1014404088 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d540042c0 con 0x7f0d58105ff0 2023-07-20T01:35:51.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.776+0000 7f0d5cf15700 1 -- 172.21.15.3:0/1014404088 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d54011a00 con 0x7f0d58105ff0 2023-07-20T01:35:51.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.776+0000 7f0d6017b700 1 -- 172.21.15.3:0/1014404088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 msgr2=0x7f0d58106410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:51.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.776+0000 7f0d6017b700 1 --2- 172.21.15.3:0/1014404088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58106410 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0d540098a0 tx=0x7f0d54009bb0 comp rx=0 tx=0).stop 2023-07-20T01:35:51.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.776+0000 7f0d6017b700 1 -- 172.21.15.3:0/1014404088 shutdown_connections 2023-07-20T01:35:51.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.776+0000 7f0d6017b700 1 --2- 172.21.15.3:0/1014404088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58106410 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:51.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.776+0000 7f0d6017b700 1 -- 172.21.15.3:0/1014404088 >> 172.21.15.3:0/1014404088 conn(0x7f0d58101590 msgr2=0x7f0d581039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:51.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.776+0000 7f0d6017b700 1 -- 172.21.15.3:0/1014404088 shutdown_connections 2023-07-20T01:35:51.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.776+0000 7f0d6017b700 1 -- 172.21.15.3:0/1014404088 wait complete. 2023-07-20T01:35:51.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.777+0000 7f0d6017b700 1 Processor -- start 2023-07-20T01:35:51.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.777+0000 7f0d6017b700 1 -- start start 2023-07-20T01:35:51.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.777+0000 7f0d6017b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:51.227 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.777+0000 7f0d6017b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5819a2e0 con 0x7f0d58105ff0 2023-07-20T01:35:51.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.777+0000 7f0d5df17700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:51.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.777+0000 7f0d5df17700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41466/0 (socket says 172.21.15.3:41466) 2023-07-20T01:35:51.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.777+0000 7f0d5df17700 1 -- 172.21.15.3:0/3183392991 learned_addr learned my addr 172.21.15.3:0/3183392991 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:51.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.778+0000 7f0d5df17700 1 -- 172.21.15.3:0/3183392991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d54009550 con 0x7f0d58105ff0 2023-07-20T01:35:51.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.778+0000 7f0d5df17700 1 --2- 172.21.15.3:0/3183392991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58199da0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009fb0 tx=0x7f0d54011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:51.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.778+0000 7f0d4effd700 1 -- 172.21.15.3:0/3183392991 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d54003a40 con 0x7f0d58105ff0 2023-07-20T01:35:51.228 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.778+0000 7f0d4effd700 1 -- 172.21.15.3:0/3183392991 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d54003ba0 con 0x7f0d58105ff0 2023-07-20T01:35:51.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.778+0000 7f0d6017b700 1 -- 172.21.15.3:0/3183392991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5819a4e0 con 0x7f0d58105ff0 2023-07-20T01:35:51.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.778+0000 7f0d4effd700 1 -- 172.21.15.3:0/3183392991 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d5401a750 con 0x7f0d58105ff0 2023-07-20T01:35:51.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.778+0000 7f0d6017b700 1 -- 172.21.15.3:0/3183392991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5819a900 con 0x7f0d58105ff0 2023-07-20T01:35:51.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.779+0000 7f0d4effd700 1 -- 172.21.15.3:0/3183392991 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0d5401ac80 con 0x7f0d58105ff0 2023-07-20T01:35:51.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.779+0000 7f0d4effd700 1 -- 172.21.15.3:0/3183392991 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0d540193c0 con 0x7f0d58105ff0 2023-07-20T01:35:51.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.779+0000 7f0d6017b700 1 -- 172.21.15.3:0/3183392991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d58193dd0 con 0x7f0d58105ff0 2023-07-20T01:35:51.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.782+0000 7f0d4effd700 1 -- 172.21.15.3:0/3183392991 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f0d5402de20 con 0x7f0d58105ff0 2023-07-20T01:35:51.229 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.915+0000 7f0d6017b700 1 -- 172.21.15.3:0/3183392991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0d5810a530 con 0x7f0d58105ff0 2023-07-20T01:35:51.230 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.915+0000 7f0d4effd700 1 -- 172.21.15.3:0/3183392991 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f0d54024b60 con 0x7f0d58105ff0 2023-07-20T01:35:51.230 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.916+0000 7f0d6017b700 1 -- 172.21.15.3:0/3183392991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 msgr2=0x7f0d58199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:51.230 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.916+0000 7f0d6017b700 1 --2- 172.21.15.3:0/3183392991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58199da0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009fb0 tx=0x7f0d54011ad0 comp rx=0 tx=0).stop 2023-07-20T01:35:51.230 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.916+0000 7f0d6017b700 1 -- 172.21.15.3:0/3183392991 shutdown_connections 2023-07-20T01:35:51.230 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.916+0000 7f0d6017b700 1 --2- 172.21.15.3:0/3183392991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d58105ff0 0x7f0d58199da0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:51.230 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.916+0000 7f0d6017b700 1 -- 172.21.15.3:0/3183392991 >> 172.21.15.3:0/3183392991 conn(0x7f0d58101590 msgr2=0x7f0d58103980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:51.230 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.916+0000 7f0d6017b700 1 -- 172.21.15.3:0/3183392991 shutdown_connections 2023-07-20T01:35:51.230 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:50.916+0000 7f0d6017b700 1 -- 172.21.15.3:0/3183392991 wait complete. 2023-07-20T01:35:51.231 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (3/15)... 2023-07-20T01:35:51.440 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:51 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:51.134+0000 7f71655a8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T01:35:52.440 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:35:52.173+0000 7f71655a8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T01:35:52.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: Activating manager daemon x 2023-07-20T01:35:52.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: mgrmap e2: x(active, starting, since 0.0033694s) 2023-07-20T01:35:52.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: from='mgr.14100 172.21.15.3:0/2276431323' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T01:35:52.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: from='mgr.14100 172.21.15.3:0/2276431323' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T01:35:52.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: from='mgr.14100 172.21.15.3:0/2276431323' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T01:35:52.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: from='mgr.14100 172.21.15.3:0/2276431323' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:35:52.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: from='mgr.14100 172.21.15.3:0/2276431323' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T01:35:52.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: Manager daemon x is now available 2023-07-20T01:35:52.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: from='mgr.14100 172.21.15.3:0/2276431323' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T01:35:52.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: from='mgr.14100 172.21.15.3:0/2276431323' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T01:35:52.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: from='mgr.14100 172.21.15.3:0/2276431323' entity='mgr.x' 2023-07-20T01:35:52.444 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:52 smithi003 ceph-mon[96422]: from='mgr.14100 172.21.15.3:0/2276431323' entity='mgr.x' 2023-07-20T01:35:53.473 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:53 smithi003 ceph-mon[96422]: from='mgr.14100 172.21.15.3:0/2276431323' entity='mgr.x' 2023-07-20T01:35:53.473 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:53 smithi003 ceph-mon[96422]: mgrmap e3: x(active, since 1.00704s) 2023-07-20T01:35:54.667 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:35:54.667 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:35:54.668 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "9b4fd2fc-269d-11ee-9b34-001a4aab830c", 2023-07-20T01:35:54.668 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T01:35:54.668 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T01:35:54.668 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T01:35:54.668 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T01:35:54.668 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:54.669 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T01:35:54.669 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T01:35:54.669 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2023-07-20T01:35:54.669 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:54.669 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T01:35:54.669 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T01:35:54.669 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:54.670 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2023-07-20T01:35:54.670 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T01:35:54.670 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:54.670 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T01:35:54.670 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T01:35:54.670 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:54.670 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T01:35:54.670 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:54.671 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T01:35:54.671 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T01:35:54.671 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T01:35:54.677 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T01:35:54.678 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T01:35:54.678 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T01:35:54.678 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:54.679 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T01:35:54.679 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T01:35:54.679 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T01:35:54.679 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T01:35:54.680 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T01:35:54.680 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T01:35:54.680 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T01:35:54.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T01:35:54.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T01:35:54.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:54.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T01:35:54.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:54.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T01:35:54.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T01:35:54.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:54.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T01:35:54.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T01:35:54.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T01:35:54.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T01:35:54.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T01:35:54.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T01:35:54.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T01:35:54.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:35:54.685 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:35:54.685 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:54.685 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T01:35:54.685 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:35:54.686 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T01:35:37.747335+0000", 2023-07-20T01:35:54.686 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:35:54.686 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:35:54.686 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T01:35:54.686 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:35:54.687 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.754+0000 7f69de2b2700 1 Processor -- start 2023-07-20T01:35:54.687 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.754+0000 7f69de2b2700 1 -- start start 2023-07-20T01:35:54.687 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.754+0000 7f69de2b2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8108eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:54.688 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.754+0000 7f69de2b2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d8073f30 con 0x7f69d8106ac0 2023-07-20T01:35:54.688 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.754+0000 7f69d7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8108eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:54.688 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.754+0000 7f69d7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8108eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41492/0 (socket says 172.21.15.3:41492) 2023-07-20T01:35:54.688 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.754+0000 7f69d7fff700 1 -- 172.21.15.3:0/2372931386 learned_addr learned my addr 172.21.15.3:0/2372931386 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:54.689 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.754+0000 7f69d7fff700 1 -- 172.21.15.3:0/2372931386 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d81093f0 con 0x7f69d8106ac0 2023-07-20T01:35:54.689 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.755+0000 7f69d7fff700 1 --2- 172.21.15.3:0/2372931386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8108eb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f69c80098a0 tx=0x7f69c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5a62dc72b8ce91 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:54.689 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.755+0000 7f69d6ffd700 1 -- 172.21.15.3:0/2372931386 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69c8004160 con 0x7f69d8106ac0 2023-07-20T01:35:54.689 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.755+0000 7f69d6ffd700 1 -- 172.21.15.3:0/2372931386 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69c80042c0 con 0x7f69d8106ac0 2023-07-20T01:35:54.689 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.755+0000 7f69d6ffd700 1 -- 172.21.15.3:0/2372931386 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69c8011a00 con 0x7f69d8106ac0 2023-07-20T01:35:54.690 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.755+0000 7f69de2b2700 1 -- 172.21.15.3:0/2372931386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 msgr2=0x7f69d8108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:54.690 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.755+0000 7f69de2b2700 1 --2- 172.21.15.3:0/2372931386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8108eb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f69c80098a0 tx=0x7f69c8009bb0 comp rx=0 tx=0).stop 2023-07-20T01:35:54.690 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.755+0000 7f69de2b2700 1 -- 172.21.15.3:0/2372931386 shutdown_connections 2023-07-20T01:35:54.690 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.755+0000 7f69de2b2700 1 --2- 172.21.15.3:0/2372931386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8108eb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:54.691 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.755+0000 7f69de2b2700 1 -- 172.21.15.3:0/2372931386 >> 172.21.15.3:0/2372931386 conn(0x7f69d8100490 msgr2=0x7f69d81028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:54.691 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.755+0000 7f69de2b2700 1 -- 172.21.15.3:0/2372931386 shutdown_connections 2023-07-20T01:35:54.691 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.756+0000 7f69de2b2700 1 -- 172.21.15.3:0/2372931386 wait complete. 2023-07-20T01:35:54.691 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.756+0000 7f69de2b2700 1 Processor -- start 2023-07-20T01:35:54.691 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.756+0000 7f69de2b2700 1 -- start start 2023-07-20T01:35:54.692 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.757+0000 7f69de2b2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8106420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:54.692 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.757+0000 7f69de2b2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d8106960 con 0x7f69d8106ac0 2023-07-20T01:35:54.692 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.757+0000 7f69d7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8106420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:54.692 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.757+0000 7f69d7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8106420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41494/0 (socket says 172.21.15.3:41494) 2023-07-20T01:35:54.693 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.757+0000 7f69d7fff700 1 -- 172.21.15.3:0/2713537595 learned_addr learned my addr 172.21.15.3:0/2713537595 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:35:54.693 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.757+0000 7f69d7fff700 1 -- 172.21.15.3:0/2713537595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c8009550 con 0x7f69d8106ac0 2023-07-20T01:35:54.693 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.757+0000 7f69d7fff700 1 --2- 172.21.15.3:0/2713537595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8106420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f69c8006af0 tx=0x7f69c8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:54.694 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.758+0000 7f69d57fa700 1 -- 172.21.15.3:0/2713537595 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69c8003a40 con 0x7f69d8106ac0 2023-07-20T01:35:54.694 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.758+0000 7f69d57fa700 1 -- 172.21.15.3:0/2713537595 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69c8003ba0 con 0x7f69d8106ac0 2023-07-20T01:35:54.694 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.758+0000 7f69de2b2700 1 -- 172.21.15.3:0/2713537595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d8104a70 con 0x7f69d8106ac0 2023-07-20T01:35:54.694 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.758+0000 7f69d57fa700 1 -- 172.21.15.3:0/2713537595 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69c801a6e0 con 0x7f69d8106ac0 2023-07-20T01:35:54.695 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.758+0000 7f69de2b2700 1 -- 172.21.15.3:0/2713537595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d8104e90 con 0x7f69d8106ac0 2023-07-20T01:35:54.695 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.759+0000 7f69d57fa700 1 -- 172.21.15.3:0/2713537595 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 3) v1 ==== 45366+0+0 (secure 0 0 0) 0x7f69c801a860 con 0x7f69d8106ac0 2023-07-20T01:35:54.695 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.759+0000 7f69d57fa700 1 --2- 172.21.15.3:0/2713537595 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f69c00385e0 0x7f69c003aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:35:54.695 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.759+0000 7f69d57fa700 1 -- 172.21.15.3:0/2713537595 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f69c804c610 con 0x7f69d8106ac0 2023-07-20T01:35:54.696 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.759+0000 7f69de2b2700 1 -- 172.21.15.3:0/2713537595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d810a140 con 0x7f69d8106ac0 2023-07-20T01:35:54.696 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.759+0000 7f69d77fe700 1 --2- 172.21.15.3:0/2713537595 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f69c00385e0 0x7f69c003aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:35:54.696 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.759+0000 7f69d77fe700 1 --2- 172.21.15.3:0/2713537595 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f69c00385e0 0x7f69c003aaa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006fd0 tx=0x7f69cc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:35:54.697 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:53.764+0000 7f69d57fa700 1 -- 172.21.15.3:0/2713537595 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f69c8018530 con 0x7f69d8106ac0 2023-07-20T01:35:54.697 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.251+0000 7f69d57fa700 1 -- 172.21.15.3:0/2713537595 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f69c802d3f0 con 0x7f69d8106ac0 2023-07-20T01:35:54.697 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.277+0000 7f69de2b2700 1 -- 172.21.15.3:0/2713537595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f69d810a360 con 0x7f69d8106ac0 2023-07-20T01:35:54.697 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.277+0000 7f69d57fa700 1 -- 172.21.15.3:0/2713537595 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f69c8018920 con 0x7f69d8106ac0 2023-07-20T01:35:54.698 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.280+0000 7f69de2b2700 1 -- 172.21.15.3:0/2713537595 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f69c00385e0 msgr2=0x7f69c003aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:54.698 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.280+0000 7f69de2b2700 1 --2- 172.21.15.3:0/2713537595 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f69c00385e0 0x7f69c003aaa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006fd0 tx=0x7f69cc006e40 comp rx=0 tx=0).stop 2023-07-20T01:35:54.698 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.280+0000 7f69de2b2700 1 -- 172.21.15.3:0/2713537595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 msgr2=0x7f69d8106420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:35:54.699 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.280+0000 7f69de2b2700 1 --2- 172.21.15.3:0/2713537595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8106420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f69c8006af0 tx=0x7f69c8011ad0 comp rx=0 tx=0).stop 2023-07-20T01:35:54.699 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.280+0000 7f69de2b2700 1 -- 172.21.15.3:0/2713537595 shutdown_connections 2023-07-20T01:35:54.699 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.280+0000 7f69de2b2700 1 --2- 172.21.15.3:0/2713537595 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f69c00385e0 0x7f69c003aaa0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:54.699 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.280+0000 7f69de2b2700 1 --2- 172.21.15.3:0/2713537595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69d8106ac0 0x7f69d8106420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:35:54.700 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.280+0000 7f69de2b2700 1 -- 172.21.15.3:0/2713537595 >> 172.21.15.3:0/2713537595 conn(0x7f69d8100490 msgr2=0x7f69d8101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:35:54.700 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.280+0000 7f69de2b2700 1 -- 172.21.15.3:0/2713537595 shutdown_connections 2023-07-20T01:35:54.700 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:54.280+0000 7f69de2b2700 1 -- 172.21.15.3:0/2713537595 wait complete. 2023-07-20T01:35:54.701 INFO:teuthology.orchestra.run.smithi003.stdout:mgr is available 2023-07-20T01:35:55.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:55 smithi003 ceph-mon[96422]: mgrmap e4: x(active, since 2s) 2023-07-20T01:35:55.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:55 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2713537595' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T01:35:57.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:35:57 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/4206498528' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T01:36:00.023 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:36:00.028 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T01:36:00.028 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:36:00.028 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T01:36:00.028 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T01:36:00.028 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T01:36:00.028 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T01:36:00.029 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:36:00.029 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T01:36:00.029 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T01:36:00.029 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-07-20T01:36:00.029 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T01:36:00.029 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T01:36:00.029 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T01:36:00.030 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T01:36:00.030 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T01:36:00.030 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T01:36:00.030 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.730+0000 7fe50d26b700 1 Processor -- start 2023-07-20T01:36:00.030 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.730+0000 7fe50d26b700 1 -- start start 2023-07-20T01:36:00.030 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.730+0000 7fe50d26b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508108e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:00.030 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.730+0000 7fe50d26b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe508079e10 con 0x7fe508106a80 2023-07-20T01:36:00.030 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.731+0000 7fe506ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508108e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:00.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.731+0000 7fe506ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508108e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41498/0 (socket says 172.21.15.3:41498) 2023-07-20T01:36:00.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.731+0000 7fe506ffd700 1 -- 172.21.15.3:0/2394484524 learned_addr learned my addr 172.21.15.3:0/2394484524 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:00.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.731+0000 7fe506ffd700 1 -- 172.21.15.3:0/2394484524 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5081093b0 con 0x7fe508106a80 2023-07-20T01:36:00.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.731+0000 7fe506ffd700 1 --2- 172.21.15.3:0/2394484524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508108e70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0098a0 tx=0x7fe4fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0c22fdcf6fec0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:00.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.731+0000 7fe505ffb700 1 -- 172.21.15.3:0/2394484524 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4fc004160 con 0x7fe508106a80 2023-07-20T01:36:00.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.732+0000 7fe505ffb700 1 -- 172.21.15.3:0/2394484524 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4fc0042c0 con 0x7fe508106a80 2023-07-20T01:36:00.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.732+0000 7fe505ffb700 1 -- 172.21.15.3:0/2394484524 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4fc011a00 con 0x7fe508106a80 2023-07-20T01:36:00.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.732+0000 7fe50d26b700 1 -- 172.21.15.3:0/2394484524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 msgr2=0x7fe508108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:00.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.732+0000 7fe50d26b700 1 --2- 172.21.15.3:0/2394484524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508108e70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0098a0 tx=0x7fe4fc009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:00.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.732+0000 7fe50d26b700 1 -- 172.21.15.3:0/2394484524 shutdown_connections 2023-07-20T01:36:00.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.732+0000 7fe50d26b700 1 --2- 172.21.15.3:0/2394484524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508108e70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:00.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.732+0000 7fe50d26b700 1 -- 172.21.15.3:0/2394484524 >> 172.21.15.3:0/2394484524 conn(0x7fe5081004d0 msgr2=0x7fe5081028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:00.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.732+0000 7fe50d26b700 1 -- 172.21.15.3:0/2394484524 shutdown_connections 2023-07-20T01:36:00.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.732+0000 7fe50d26b700 1 -- 172.21.15.3:0/2394484524 wait complete. 2023-07-20T01:36:00.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.733+0000 7fe50d26b700 1 Processor -- start 2023-07-20T01:36:00.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.733+0000 7fe50d26b700 1 -- start start 2023-07-20T01:36:00.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.733+0000 7fe50d26b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:00.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.733+0000 7fe50d26b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50819a280 con 0x7fe508106a80 2023-07-20T01:36:00.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.733+0000 7fe506ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:00.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.733+0000 7fe506ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41500/0 (socket says 172.21.15.3:41500) 2023-07-20T01:36:00.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.733+0000 7fe506ffd700 1 -- 172.21.15.3:0/4206498528 learned_addr learned my addr 172.21.15.3:0/4206498528 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:00.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.734+0000 7fe506ffd700 1 -- 172.21.15.3:0/4206498528 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc009550 con 0x7fe508106a80 2023-07-20T01:36:00.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.734+0000 7fe506ffd700 1 --2- 172.21.15.3:0/4206498528 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508199d40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009fb0 tx=0x7fe4fc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:00.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.734+0000 7fe4effff700 1 -- 172.21.15.3:0/4206498528 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4fc003a40 con 0x7fe508106a80 2023-07-20T01:36:00.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.734+0000 7fe4effff700 1 -- 172.21.15.3:0/4206498528 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4fc003ba0 con 0x7fe508106a80 2023-07-20T01:36:00.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.734+0000 7fe50d26b700 1 -- 172.21.15.3:0/4206498528 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe50819a480 con 0x7fe508106a80 2023-07-20T01:36:00.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.734+0000 7fe4effff700 1 -- 172.21.15.3:0/4206498528 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4fc01a750 con 0x7fe508106a80 2023-07-20T01:36:00.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.734+0000 7fe50d26b700 1 -- 172.21.15.3:0/4206498528 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50819a8a0 con 0x7fe508106a80 2023-07-20T01:36:00.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.735+0000 7fe4effff700 1 -- 172.21.15.3:0/4206498528 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fe4fc01a980 con 0x7fe508106a80 2023-07-20T01:36:00.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.735+0000 7fe4effff700 1 --2- 172.21.15.3:0/4206498528 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fe4f00386c0 0x7fe4f003ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:00.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.735+0000 7fe4effff700 1 -- 172.21.15.3:0/4206498528 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe4fc04c4e0 con 0x7fe508106a80 2023-07-20T01:36:00.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.735+0000 7fe5067fc700 1 --2- 172.21.15.3:0/4206498528 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fe4f00386c0 0x7fe4f003ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:00.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.735+0000 7fe50d26b700 1 -- 172.21.15.3:0/4206498528 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe508061da0 con 0x7fe508106a80 2023-07-20T01:36:00.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.736+0000 7fe5067fc700 1 --2- 172.21.15.3:0/4206498528 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fe4f00386c0 0x7fe4f003ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8006fd0 tx=0x7fe4f8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:00.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:56.740+0000 7fe4effff700 1 -- 172.21.15.3:0/4206498528 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe4fc01ac30 con 0x7fe508106a80 2023-07-20T01:36:00.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.095+0000 7fe50d26b700 1 -- 172.21.15.3:0/4206498528 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe50819d2e0 con 0x7fe508106a80 2023-07-20T01:36:00.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.096+0000 7fe4effff700 1 -- 172.21.15.3:0/4206498528 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fe4fc018350 con 0x7fe508106a80 2023-07-20T01:36:00.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.098+0000 7fe50d26b700 1 -- 172.21.15.3:0/4206498528 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fe4f00386c0 msgr2=0x7fe4f003ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:00.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.098+0000 7fe50d26b700 1 --2- 172.21.15.3:0/4206498528 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fe4f00386c0 0x7fe4f003ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8006fd0 tx=0x7fe4f8006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:00.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.098+0000 7fe50d26b700 1 -- 172.21.15.3:0/4206498528 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 msgr2=0x7fe508199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:00.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.098+0000 7fe50d26b700 1 --2- 172.21.15.3:0/4206498528 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508199d40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009fb0 tx=0x7fe4fc011ad0 comp rx=0 tx=0).stop 2023-07-20T01:36:00.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.098+0000 7fe50d26b700 1 -- 172.21.15.3:0/4206498528 shutdown_connections 2023-07-20T01:36:00.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.098+0000 7fe50d26b700 1 --2- 172.21.15.3:0/4206498528 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fe4f00386c0 0x7fe4f003ab80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:00.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.098+0000 7fe50d26b700 1 --2- 172.21.15.3:0/4206498528 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe508106a80 0x7fe508199d40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:00.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.098+0000 7fe50d26b700 1 -- 172.21.15.3:0/4206498528 >> 172.21.15.3:0/4206498528 conn(0x7fe5081004d0 msgr2=0x7fe508101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:00.037 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.098+0000 7fe50d26b700 1 -- 172.21.15.3:0/4206498528 shutdown_connections 2023-07-20T01:36:00.037 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:35:57.098+0000 7fe50d26b700 1 -- 172.21.15.3:0/4206498528 wait complete. 2023-07-20T01:36:00.037 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling cephadm module... 2023-07-20T01:36:02.599 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:02 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2998407323' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-20T01:36:02.599 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:02 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: ignoring --setuser ceph since I am not root 2023-07-20T01:36:02.600 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:02 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: ignoring --setgroup ceph since I am not root 2023-07-20T01:36:02.600 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:02 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:02.532+0000 7f3d5eec8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T01:36:02.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:02 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:02.627+0000 7f3d5eec8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T01:36:02.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:02 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:02.735+0000 7f3d5eec8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T01:36:02.941 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:02 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:02.846+0000 7f3d5eec8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T01:36:03.324 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:03 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:03.057+0000 7f3d5eec8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T01:36:03.681 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:03 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:03.450+0000 7f3d5eec8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T01:36:03.682 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:03 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:03.559+0000 7f3d5eec8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T01:36:03.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:03 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2998407323' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-20T01:36:03.682 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:03 smithi003 ceph-mon[96422]: mgrmap e5: x(active, since 10s) 2023-07-20T01:36:03.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:03 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:03.709+0000 7f3d5eec8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T01:36:03.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:03 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:03.825+0000 7f3d5eec8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T01:36:04.690 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:04 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:04.332+0000 7f3d5eec8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T01:36:04.794 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.816+0000 7f6b7e016700 1 Processor -- start 2023-07-20T01:36:04.795 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.816+0000 7f6b7e016700 1 -- start start 2023-07-20T01:36:04.795 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.817+0000 7f6b7e016700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b78104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:04.795 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.817+0000 7f6b7e016700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b78073f10 con 0x7f6b781048d0 2023-07-20T01:36:04.795 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.817+0000 7f6b777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b78104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:04.795 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.817+0000 7f6b777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b78104cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41504/0 (socket says 172.21.15.3:41504) 2023-07-20T01:36:04.795 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.817+0000 7f6b777fe700 1 -- 172.21.15.3:0/3870013217 learned_addr learned my addr 172.21.15.3:0/3870013217 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:04.795 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.817+0000 7f6b777fe700 1 -- 172.21.15.3:0/3870013217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b78074050 con 0x7f6b781048d0 2023-07-20T01:36:04.796 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.817+0000 7f6b777fe700 1 --2- 172.21.15.3:0/3870013217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b78104cf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c0098a0 tx=0x7f6b6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e036647ea54d670 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:04.796 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.818+0000 7f6b767fc700 1 -- 172.21.15.3:0/3870013217 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b6c004160 con 0x7f6b781048d0 2023-07-20T01:36:04.796 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.818+0000 7f6b767fc700 1 -- 172.21.15.3:0/3870013217 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b6c0042c0 con 0x7f6b781048d0 2023-07-20T01:36:04.796 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.818+0000 7f6b767fc700 1 -- 172.21.15.3:0/3870013217 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b6c011a00 con 0x7f6b781048d0 2023-07-20T01:36:04.796 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.818+0000 7f6b7e016700 1 -- 172.21.15.3:0/3870013217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 msgr2=0x7f6b78104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:04.796 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.818+0000 7f6b7e016700 1 --2- 172.21.15.3:0/3870013217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b78104cf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c0098a0 tx=0x7f6b6c009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:04.797 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.818+0000 7f6b7e016700 1 -- 172.21.15.3:0/3870013217 shutdown_connections 2023-07-20T01:36:04.797 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.818+0000 7f6b7e016700 1 --2- 172.21.15.3:0/3870013217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b78104cf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:04.797 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.818+0000 7f6b7e016700 1 -- 172.21.15.3:0/3870013217 >> 172.21.15.3:0/3870013217 conn(0x7f6b78100490 msgr2=0x7f6b781028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:04.797 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.818+0000 7f6b7e016700 1 -- 172.21.15.3:0/3870013217 shutdown_connections 2023-07-20T01:36:04.797 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.818+0000 7f6b7e016700 1 -- 172.21.15.3:0/3870013217 wait complete. 2023-07-20T01:36:04.797 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.819+0000 7f6b7e016700 1 Processor -- start 2023-07-20T01:36:04.797 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.819+0000 7f6b7e016700 1 -- start start 2023-07-20T01:36:04.798 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b7e016700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b7819bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:04.798 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b7e016700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7819c420 con 0x7f6b781048d0 2023-07-20T01:36:04.798 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b7819bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:04.798 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b7819bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41506/0 (socket says 172.21.15.3:41506) 2023-07-20T01:36:04.798 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b777fe700 1 -- 172.21.15.3:0/2998407323 learned_addr learned my addr 172.21.15.3:0/2998407323 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:04.798 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b777fe700 1 -- 172.21.15.3:0/2998407323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c009550 con 0x7f6b781048d0 2023-07-20T01:36:04.798 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b777fe700 1 --2- 172.21.15.3:0/2998407323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b7819bee0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c006af0 tx=0x7f6b6c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:04.799 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b74ff9700 1 -- 172.21.15.3:0/2998407323 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b6c003a40 con 0x7f6b781048d0 2023-07-20T01:36:04.799 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b74ff9700 1 -- 172.21.15.3:0/2998407323 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b6c003ba0 con 0x7f6b781048d0 2023-07-20T01:36:04.799 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b74ff9700 1 -- 172.21.15.3:0/2998407323 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b6c01a6e0 con 0x7f6b781048d0 2023-07-20T01:36:04.799 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b7e016700 1 -- 172.21.15.3:0/2998407323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7819c620 con 0x7f6b781048d0 2023-07-20T01:36:04.799 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.820+0000 7f6b7e016700 1 -- 172.21.15.3:0/2998407323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7819ca40 con 0x7f6b781048d0 2023-07-20T01:36:04.799 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.821+0000 7f6b74ff9700 1 -- 172.21.15.3:0/2998407323 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f6b6c024430 con 0x7f6b781048d0 2023-07-20T01:36:04.799 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.822+0000 7f6b74ff9700 1 --2- 172.21.15.3:0/2998407323 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f6b60038710 0x7f6b6003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:04.800 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.822+0000 7f6b74ff9700 1 -- 172.21.15.3:0/2998407323 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6b6c04c660 con 0x7f6b781048d0 2023-07-20T01:36:04.800 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.822+0000 7f6b7e016700 1 -- 172.21.15.3:0/2998407323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b78195760 con 0x7f6b781048d0 2023-07-20T01:36:04.800 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.822+0000 7f6b76ffd700 1 --2- 172.21.15.3:0/2998407323 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f6b60038710 0x7f6b6003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:04.800 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.822+0000 7f6b76ffd700 1 --2- 172.21.15.3:0/2998407323 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f6b60038710 0x7f6b6003abd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6b68006fd0 tx=0x7f6b68006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:04.800 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:01.826+0000 7f6b74ff9700 1 -- 172.21.15.3:0/2998407323 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6b6c0246e0 con 0x7f6b781048d0 2023-07-20T01:36:04.800 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.295+0000 7f6b7e016700 1 -- 172.21.15.3:0/2998407323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f6b78061da0 con 0x7f6b781048d0 2023-07-20T01:36:04.800 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.347+0000 7f6b74ff9700 1 -- 172.21.15.3:0/2998407323 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f6b6c024a60 con 0x7f6b781048d0 2023-07-20T01:36:04.801 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.349+0000 7f6b74ff9700 1 -- 172.21.15.3:0/2998407323 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f6b6c0246e0 con 0x7f6b781048d0 2023-07-20T01:36:04.801 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.352+0000 7f6b7e016700 1 -- 172.21.15.3:0/2998407323 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f6b60038710 msgr2=0x7f6b6003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:04.801 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.352+0000 7f6b7e016700 1 --2- 172.21.15.3:0/2998407323 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f6b60038710 0x7f6b6003abd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6b68006fd0 tx=0x7f6b68006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:04.801 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.352+0000 7f6b7e016700 1 -- 172.21.15.3:0/2998407323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 msgr2=0x7f6b7819bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:04.801 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.352+0000 7f6b7e016700 1 --2- 172.21.15.3:0/2998407323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b7819bee0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c006af0 tx=0x7f6b6c011ad0 comp rx=0 tx=0).stop 2023-07-20T01:36:04.801 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.352+0000 7f6b7e016700 1 -- 172.21.15.3:0/2998407323 shutdown_connections 2023-07-20T01:36:04.801 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.352+0000 7f6b7e016700 1 --2- 172.21.15.3:0/2998407323 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f6b60038710 0x7f6b6003abd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:04.802 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.352+0000 7f6b7e016700 1 --2- 172.21.15.3:0/2998407323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b781048d0 0x7f6b7819bee0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:04.802 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.352+0000 7f6b7e016700 1 -- 172.21.15.3:0/2998407323 >> 172.21.15.3:0/2998407323 conn(0x7f6b78100490 msgr2=0x7f6b78190490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:04.802 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.352+0000 7f6b7e016700 1 -- 172.21.15.3:0/2998407323 shutdown_connections 2023-07-20T01:36:04.802 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:02.352+0000 7f6b7e016700 1 -- 172.21.15.3:0/2998407323 wait complete. 2023-07-20T01:36:05.054 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:05 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:05.021+0000 7f3d5eec8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T01:36:05.440 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:05 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:05.214+0000 7f3d5eec8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T01:36:06.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:05 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2599870930' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T01:36:06.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:36:06.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-20T01:36:06.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T01:36:06.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T01:36:06.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T01:36:06.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:36:06.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.392+0000 7f002a1ce700 1 Processor -- start 2023-07-20T01:36:06.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.392+0000 7f002a1ce700 1 -- start start 2023-07-20T01:36:06.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.392+0000 7f002a1ce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024108f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:06.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.392+0000 7f002a1ce700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0024073ed0 con 0x7f0024106b20 2023-07-20T01:36:06.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.392+0000 7f00237fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024108f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:06.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.392+0000 7f00237fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024108f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41524/0 (socket says 172.21.15.3:41524) 2023-07-20T01:36:06.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.392+0000 7f00237fe700 1 -- 172.21.15.3:0/1700088227 learned_addr learned my addr 172.21.15.3:0/1700088227 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:06.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.392+0000 7f00237fe700 1 -- 172.21.15.3:0/1700088227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0024074010 con 0x7f0024106b20 2023-07-20T01:36:06.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.393+0000 7f00237fe700 1 --2- 172.21.15.3:0/1700088227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024108f10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f00140098a0 tx=0x7f0014009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716108066d2961ec server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:06.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.393+0000 7f00227fc700 1 -- 172.21.15.3:0/1700088227 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0014004160 con 0x7f0024106b20 2023-07-20T01:36:06.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.393+0000 7f00227fc700 1 -- 172.21.15.3:0/1700088227 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00140042c0 con 0x7f0024106b20 2023-07-20T01:36:06.176 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.393+0000 7f00227fc700 1 -- 172.21.15.3:0/1700088227 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0014011a00 con 0x7f0024106b20 2023-07-20T01:36:06.176 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.394+0000 7f002a1ce700 1 -- 172.21.15.3:0/1700088227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 msgr2=0x7f0024108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:06.176 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.394+0000 7f002a1ce700 1 --2- 172.21.15.3:0/1700088227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024108f10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f00140098a0 tx=0x7f0014009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:06.176 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.394+0000 7f002a1ce700 1 -- 172.21.15.3:0/1700088227 shutdown_connections 2023-07-20T01:36:06.176 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.394+0000 7f002a1ce700 1 --2- 172.21.15.3:0/1700088227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024108f10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:06.176 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.394+0000 7f002a1ce700 1 -- 172.21.15.3:0/1700088227 >> 172.21.15.3:0/1700088227 conn(0x7f0024100490 msgr2=0x7f00241028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:06.176 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.394+0000 7f002a1ce700 1 -- 172.21.15.3:0/1700088227 shutdown_connections 2023-07-20T01:36:06.176 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.394+0000 7f002a1ce700 1 -- 172.21.15.3:0/1700088227 wait complete. 2023-07-20T01:36:06.177 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.394+0000 7f002a1ce700 1 Processor -- start 2023-07-20T01:36:06.177 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.395+0000 7f002a1ce700 1 -- start start 2023-07-20T01:36:06.177 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.395+0000 7f002a1ce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:06.177 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.395+0000 7f002a1ce700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002419a240 con 0x7f0024106b20 2023-07-20T01:36:06.177 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.395+0000 7f00237fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:06.177 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.395+0000 7f00237fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41526/0 (socket says 172.21.15.3:41526) 2023-07-20T01:36:06.177 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.395+0000 7f00237fe700 1 -- 172.21.15.3:0/2599870930 learned_addr learned my addr 172.21.15.3:0/2599870930 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:06.178 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.395+0000 7f00237fe700 1 -- 172.21.15.3:0/2599870930 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0014009550 con 0x7f0024106b20 2023-07-20T01:36:06.178 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.395+0000 7f00237fe700 1 --2- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024199d00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0014006af0 tx=0x7f0014011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:06.178 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.395+0000 7f0020ff9700 1 -- 172.21.15.3:0/2599870930 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0014003a40 con 0x7f0024106b20 2023-07-20T01:36:06.178 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.396+0000 7f0020ff9700 1 -- 172.21.15.3:0/2599870930 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0014003ba0 con 0x7f0024106b20 2023-07-20T01:36:06.178 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.396+0000 7f0020ff9700 1 -- 172.21.15.3:0/2599870930 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f001401a6e0 con 0x7f0024106b20 2023-07-20T01:36:06.178 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.396+0000 7f002a1ce700 1 -- 172.21.15.3:0/2599870930 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002419a440 con 0x7f0024106b20 2023-07-20T01:36:06.178 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.396+0000 7f002a1ce700 1 -- 172.21.15.3:0/2599870930 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002419a860 con 0x7f0024106b20 2023-07-20T01:36:06.178 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.397+0000 7f0020ff9700 1 -- 172.21.15.3:0/2599870930 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f001401a860 con 0x7f0024106b20 2023-07-20T01:36:06.179 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.397+0000 7f002a1ce700 1 -- 172.21.15.3:0/2599870930 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0024061da0 con 0x7f0024106b20 2023-07-20T01:36:06.179 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.397+0000 7f0020ff9700 1 --2- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f000c038760 0x7f000c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:06.179 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.397+0000 7f0020ff9700 1 -- 172.21.15.3:0/2599870930 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f001404c6e0 con 0x7f0024106b20 2023-07-20T01:36:06.179 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.397+0000 7f0022ffd700 1 -- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f000c038760 msgr2=0x7f000c03ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/417953684 2023-07-20T01:36:06.179 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.397+0000 7f0022ffd700 1 --2- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f000c038760 0x7f000c03ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:36:06.179 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.402+0000 7f0020ff9700 1 -- 172.21.15.3:0/2599870930 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f001401ab10 con 0x7f0024106b20 2023-07-20T01:36:06.179 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.598+0000 7f0022ffd700 1 -- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f000c038760 msgr2=0x7f000c03ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/417953684 2023-07-20T01:36:06.180 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.598+0000 7f0022ffd700 1 --2- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f000c038760 0x7f000c03ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T01:36:06.180 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.868+0000 7f002a1ce700 1 -- 172.21.15.3:0/2599870930 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f002419d3f0 con 0x7f0024106b20 2023-07-20T01:36:06.180 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.869+0000 7f0020ff9700 1 -- 172.21.15.3:0/2599870930 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f0014018350 con 0x7f0024106b20 2023-07-20T01:36:06.180 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.871+0000 7f002a1ce700 1 -- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f000c038760 msgr2=0x7f000c03ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:36:06.180 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.871+0000 7f002a1ce700 1 --2- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f000c038760 0x7f000c03ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:06.180 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.871+0000 7f002a1ce700 1 -- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 msgr2=0x7f0024199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:06.180 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.871+0000 7f002a1ce700 1 --2- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024199d00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0014006af0 tx=0x7f0014011ad0 comp rx=0 tx=0).stop 2023-07-20T01:36:06.180 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.871+0000 7f002a1ce700 1 -- 172.21.15.3:0/2599870930 shutdown_connections 2023-07-20T01:36:06.181 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.871+0000 7f002a1ce700 1 --2- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7f000c038760 0x7f000c03ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:06.181 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.871+0000 7f002a1ce700 1 --2- 172.21.15.3:0/2599870930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0024106b20 0x7f0024199d00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:06.181 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.871+0000 7f002a1ce700 1 -- 172.21.15.3:0/2599870930 >> 172.21.15.3:0/2599870930 conn(0x7f0024100490 msgr2=0x7f0024101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:06.181 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.871+0000 7f002a1ce700 1 -- 172.21.15.3:0/2599870930 shutdown_connections 2023-07-20T01:36:06.181 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:05.871+0000 7f002a1ce700 1 -- 172.21.15.3:0/2599870930 wait complete. 2023-07-20T01:36:06.182 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2023-07-20T01:36:06.182 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 5... 2023-07-20T01:36:06.433 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:06 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:06.190+0000 7f3d5eec8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:36:06.434 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:06 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:06.306+0000 7f3d5eec8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T01:36:06.690 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:06 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:06.531+0000 7f3d5eec8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:36:07.121 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:06 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:06.884+0000 7f3d5eec8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T01:36:07.440 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:07 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:07.149+0000 7f3d5eec8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T01:36:07.440 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:07 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:07.331+0000 7f3d5eec8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T01:36:07.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:07 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:07.657+0000 7f3d5eec8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T01:36:08.852 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:08 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:08.481+0000 7f3d5eec8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T01:36:08.852 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:08 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:08.637+0000 7f3d5eec8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T01:36:09.190 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:08 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:08.880+0000 7f3d5eec8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T01:36:09.190 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:09 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:09.015+0000 7f3d5eec8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T01:36:09.690 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:09 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:09.284+0000 7f3d5eec8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T01:36:10.690 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:10.323+0000 7f3d5eec8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T01:36:10.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: Active manager daemon x restarted 2023-07-20T01:36:10.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: Activating manager daemon x 2023-07-20T01:36:10.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: osdmap e2: 0 total, 0 up, 0 in 2023-07-20T01:36:10.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: mgrmap e6: x(active, starting, since 0.00362838s) 2023-07-20T01:36:10.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:36:10.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T01:36:10.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T01:36:10.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T01:36:10.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T01:36:10.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: Manager daemon x is now available 2023-07-20T01:36:10.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:10.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:10.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:10 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:36:11.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:11 smithi003 ceph-mon[96422]: Found migration_current of "None". Setting to last migration. 2023-07-20T01:36:11.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:11 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:36:11.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:11 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:36:11.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:11 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T01:36:11.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:11 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T01:36:11.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:11 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:11.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:11 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:11.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:11 smithi003 ceph-mon[96422]: mgrmap e7: x(active, since 1.00568s) 2023-07-20T01:36:11.729 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:36:11.729 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-20T01:36:11.729 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T01:36:11.729 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:36:11.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.715+0000 7fcd5b770700 1 Processor -- start 2023-07-20T01:36:11.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.715+0000 7fcd5b770700 1 -- start start 2023-07-20T01:36:11.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.715+0000 7fcd5b770700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54108e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:11.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.715+0000 7fcd5b770700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd54073fb0 con 0x7fcd54106a40 2023-07-20T01:36:11.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.716+0000 7fcd5950c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54108e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:11.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.716+0000 7fcd5950c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54108e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41532/0 (socket says 172.21.15.3:41532) 2023-07-20T01:36:11.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.716+0000 7fcd5950c700 1 -- 172.21.15.3:0/2864392747 learned_addr learned my addr 172.21.15.3:0/2864392747 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:11.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.716+0000 7fcd5950c700 1 -- 172.21.15.3:0/2864392747 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd54109370 con 0x7fcd54106a40 2023-07-20T01:36:11.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.716+0000 7fcd5950c700 1 --2- 172.21.15.3:0/2864392747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54108e30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcd500098a0 tx=0x7fcd50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2408f0bffdc87fc3 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:11.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.716+0000 7fcd4bfff700 1 -- 172.21.15.3:0/2864392747 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd50004160 con 0x7fcd54106a40 2023-07-20T01:36:11.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.716+0000 7fcd4bfff700 1 -- 172.21.15.3:0/2864392747 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd500042c0 con 0x7fcd54106a40 2023-07-20T01:36:11.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.716+0000 7fcd4bfff700 1 -- 172.21.15.3:0/2864392747 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd50011a00 con 0x7fcd54106a40 2023-07-20T01:36:11.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.717+0000 7fcd5b770700 1 -- 172.21.15.3:0/2864392747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 msgr2=0x7fcd54108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:11.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.717+0000 7fcd5b770700 1 --2- 172.21.15.3:0/2864392747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54108e30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcd500098a0 tx=0x7fcd50009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:11.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.717+0000 7fcd5b770700 1 -- 172.21.15.3:0/2864392747 shutdown_connections 2023-07-20T01:36:11.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.717+0000 7fcd5b770700 1 --2- 172.21.15.3:0/2864392747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54108e30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:11.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.717+0000 7fcd5b770700 1 -- 172.21.15.3:0/2864392747 >> 172.21.15.3:0/2864392747 conn(0x7fcd54100490 msgr2=0x7fcd541028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:11.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.717+0000 7fcd5b770700 1 -- 172.21.15.3:0/2864392747 shutdown_connections 2023-07-20T01:36:11.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.717+0000 7fcd5b770700 1 -- 172.21.15.3:0/2864392747 wait complete. 2023-07-20T01:36:11.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.718+0000 7fcd5b770700 1 Processor -- start 2023-07-20T01:36:11.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.718+0000 7fcd5b770700 1 -- start start 2023-07-20T01:36:11.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.718+0000 7fcd5b770700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:11.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.718+0000 7fcd5b770700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5419a1a0 con 0x7fcd54106a40 2023-07-20T01:36:11.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.718+0000 7fcd5950c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:11.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.718+0000 7fcd5950c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41534/0 (socket says 172.21.15.3:41534) 2023-07-20T01:36:11.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.718+0000 7fcd5950c700 1 -- 172.21.15.3:0/4201378591 learned_addr learned my addr 172.21.15.3:0/4201378591 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:11.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.719+0000 7fcd5950c700 1 -- 172.21.15.3:0/4201378591 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd50009550 con 0x7fcd54106a40 2023-07-20T01:36:11.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.719+0000 7fcd5950c700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54199c60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcd50006af0 tx=0x7fcd50011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:11.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.719+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd50003a40 con 0x7fcd54106a40 2023-07-20T01:36:11.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.719+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd50003ba0 con 0x7fcd54106a40 2023-07-20T01:36:11.734 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.719+0000 7fcd5b770700 1 -- 172.21.15.3:0/4201378591 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd5419a3a0 con 0x7fcd54106a40 2023-07-20T01:36:11.734 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.719+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd5001a6e0 con 0x7fcd54106a40 2023-07-20T01:36:11.734 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.719+0000 7fcd5b770700 1 -- 172.21.15.3:0/4201378591 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5419a7c0 con 0x7fcd54106a40 2023-07-20T01:36:11.734 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.720+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fcd50024a60 con 0x7fcd54106a40 2023-07-20T01:36:11.734 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.720+0000 7fcd4a7fc700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 0x7fcd4003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:11.734 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.721+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 --> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd4003b330 con 0x7fcd40038760 2023-07-20T01:36:11.734 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.721+0000 7fcd58d0b700 1 -- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 msgr2=0x7fcd4003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/417953684 2023-07-20T01:36:11.735 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.721+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcd5004c6f0 con 0x7fcd54106a40 2023-07-20T01:36:11.735 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.721+0000 7fcd58d0b700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 0x7fcd4003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:36:11.735 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.922+0000 7fcd58d0b700 1 -- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 msgr2=0x7fcd4003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/417953684 2023-07-20T01:36:11.735 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:06.922+0000 7fcd58d0b700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 0x7fcd4003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T01:36:11.735 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:07.323+0000 7fcd58d0b700 1 -- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 msgr2=0x7fcd4003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/417953684 2023-07-20T01:36:11.735 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:07.323+0000 7fcd58d0b700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 0x7fcd4003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T01:36:11.736 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:08.124+0000 7fcd58d0b700 1 -- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 msgr2=0x7fcd4003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/417953684 2023-07-20T01:36:11.736 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:08.124+0000 7fcd58d0b700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 0x7fcd4003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T01:36:11.736 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:09.725+0000 7fcd58d0b700 1 -- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 msgr2=0x7fcd4003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/417953684 2023-07-20T01:36:11.736 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:09.725+0000 7fcd58d0b700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 0x7fcd4003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T01:36:11.736 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:10.338+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7fcd50024610 con 0x7fcd54106a40 2023-07-20T01:36:11.736 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:10.338+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 msgr2=0x7fcd4003ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:36:11.736 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:10.338+0000 7fcd4a7fc700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 0x7fcd4003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:11.736 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.339+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fcd5004e300 con 0x7fcd54106a40 2023-07-20T01:36:11.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.339+0000 7fcd4a7fc700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fcd4003bff0 0x7fcd4003e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:11.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.339+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 --> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd4003b330 con 0x7fcd4003bff0 2023-07-20T01:36:11.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.340+0000 7fcd58d0b700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fcd4003bff0 0x7fcd4003e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:11.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.340+0000 7fcd58d0b700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fcd4003bff0 0x7fcd4003e3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcd44003a10 tx=0x7fcd440092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:11.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.341+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 <== mgr.14120 v2:172.21.15.3:6800/596448098 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fcd4003b330 con 0x7fcd4003bff0 2023-07-20T01:36:11.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd5b770700 1 -- 172.21.15.3:0/4201378591 --> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcd54061da0 con 0x7fcd4003bff0 2023-07-20T01:36:11.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd4a7fc700 1 -- 172.21.15.3:0/4201378591 <== mgr.14120 v2:172.21.15.3:6800/596448098 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fcd54061da0 con 0x7fcd4003bff0 2023-07-20T01:36:11.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd5b770700 1 -- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fcd4003bff0 msgr2=0x7fcd4003e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:11.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd5b770700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fcd4003bff0 0x7fcd4003e3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcd44003a10 tx=0x7fcd440092b0 comp rx=0 tx=0).stop 2023-07-20T01:36:11.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd5b770700 1 -- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 msgr2=0x7fcd54199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:11.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd5b770700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54199c60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcd50006af0 tx=0x7fcd50011ad0 comp rx=0 tx=0).stop 2023-07-20T01:36:11.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd5b770700 1 -- 172.21.15.3:0/4201378591 shutdown_connections 2023-07-20T01:36:11.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd5b770700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fcd4003bff0 0x7fcd4003e3e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:11.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd5b770700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd54106a40 0x7fcd54199c60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:11.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd5b770700 1 --2- 172.21.15.3:0/4201378591 >> [v2:172.21.15.3:6800/417953684,v1:172.21.15.3:6801/417953684] conn(0x7fcd40038760 0x7fcd4003ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:11.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd5b770700 1 -- 172.21.15.3:0/4201378591 >> 172.21.15.3:0/4201378591 conn(0x7fcd54100490 msgr2=0x7fcd54101b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:11.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.352+0000 7fcd5b770700 1 -- 172.21.15.3:0/4201378591 shutdown_connections 2023-07-20T01:36:11.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:11.353+0000 7fcd5b770700 1 -- 172.21.15.3:0/4201378591 wait complete. 2023-07-20T01:36:11.739 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 5 is available 2023-07-20T01:36:11.739 INFO:teuthology.orchestra.run.smithi003.stdout:Setting orchestrator backend to cephadm... 2023-07-20T01:36:12.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:12 smithi003 ceph-mon[96422]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T01:36:12.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:12 smithi003 ceph-mon[96422]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T01:36:12.973 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.231+0000 7f1007cff700 1 Processor -- start 2023-07-20T01:36:12.973 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.231+0000 7f1007cff700 1 -- start start 2023-07-20T01:36:12.974 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.232+0000 7f1007cff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100007a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:12.974 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.232+0000 7f1007cff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100007a810 con 0x7f100007bde0 2023-07-20T01:36:12.974 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.232+0000 7f1005a9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100007a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:12.974 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.232+0000 7f1005a9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100007a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41576/0 (socket says 172.21.15.3:41576) 2023-07-20T01:36:12.974 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.232+0000 7f1005a9b700 1 -- 172.21.15.3:0/113272471 learned_addr learned my addr 172.21.15.3:0/113272471 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:12.974 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.232+0000 7f1005a9b700 1 -- 172.21.15.3:0/113272471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100007a950 con 0x7f100007bde0 2023-07-20T01:36:12.974 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.232+0000 7f1005a9b700 1 --2- 172.21.15.3:0/113272471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100007a240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0098a0 tx=0x7f0ffc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ef1daa094e40eb server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:12.975 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.232+0000 7f1004a99700 1 -- 172.21.15.3:0/113272471 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ffc004160 con 0x7f100007bde0 2023-07-20T01:36:12.975 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.232+0000 7f1004a99700 1 -- 172.21.15.3:0/113272471 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ffc0042c0 con 0x7f100007bde0 2023-07-20T01:36:12.975 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.232+0000 7f1004a99700 1 -- 172.21.15.3:0/113272471 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ffc011a00 con 0x7f100007bde0 2023-07-20T01:36:12.975 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.233+0000 7f1007cff700 1 -- 172.21.15.3:0/113272471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 msgr2=0x7f100007a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:12.975 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.233+0000 7f1007cff700 1 --2- 172.21.15.3:0/113272471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100007a240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0098a0 tx=0x7f0ffc009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:12.975 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.233+0000 7f1007cff700 1 -- 172.21.15.3:0/113272471 shutdown_connections 2023-07-20T01:36:12.976 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.233+0000 7f1007cff700 1 --2- 172.21.15.3:0/113272471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100007a240 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:12.976 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.233+0000 7f1007cff700 1 -- 172.21.15.3:0/113272471 >> 172.21.15.3:0/113272471 conn(0x7f10001015a0 msgr2=0x7f10001039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:12.976 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.233+0000 7f1007cff700 1 -- 172.21.15.3:0/113272471 shutdown_connections 2023-07-20T01:36:12.976 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.233+0000 7f1007cff700 1 -- 172.21.15.3:0/113272471 wait complete. 2023-07-20T01:36:12.976 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.234+0000 7f1007cff700 1 Processor -- start 2023-07-20T01:36:12.976 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.234+0000 7f1007cff700 1 -- start start 2023-07-20T01:36:12.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.234+0000 7f1007cff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100019c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:12.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.234+0000 7f1007cff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100019c770 con 0x7f100007bde0 2023-07-20T01:36:12.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.234+0000 7f1005a9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100019c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:12.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.234+0000 7f1005a9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100019c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41578/0 (socket says 172.21.15.3:41578) 2023-07-20T01:36:12.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.234+0000 7f1005a9b700 1 -- 172.21.15.3:0/885086731 learned_addr learned my addr 172.21.15.3:0/885086731 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:12.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.234+0000 7f1005a9b700 1 -- 172.21.15.3:0/885086731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc009550 con 0x7f100007bde0 2023-07-20T01:36:12.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.235+0000 7f1005a9b700 1 --2- 172.21.15.3:0/885086731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100019c230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009130 tx=0x7f0ffc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:12.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.235+0000 7f0ff6ffd700 1 -- 172.21.15.3:0/885086731 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ffc003b70 con 0x7f100007bde0 2023-07-20T01:36:12.978 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.235+0000 7f0ff6ffd700 1 -- 172.21.15.3:0/885086731 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ffc003cd0 con 0x7f100007bde0 2023-07-20T01:36:12.978 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.235+0000 7f1007cff700 1 -- 172.21.15.3:0/885086731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100019c970 con 0x7f100007bde0 2023-07-20T01:36:12.978 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.235+0000 7f0ff6ffd700 1 -- 172.21.15.3:0/885086731 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ffc0197d0 con 0x7f100007bde0 2023-07-20T01:36:12.978 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.235+0000 7f1007cff700 1 -- 172.21.15.3:0/885086731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100019cd90 con 0x7f100007bde0 2023-07-20T01:36:12.979 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.236+0000 7f0ff6ffd700 1 -- 172.21.15.3:0/885086731 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f0ffc0199c0 con 0x7f100007bde0 2023-07-20T01:36:12.979 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.236+0000 7f0ff6ffd700 1 --2- 172.21.15.3:0/885086731 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0fec038630 0x7f0fec03aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:12.979 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.236+0000 7f0ff6ffd700 1 -- 172.21.15.3:0/885086731 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0ffc04c6e0 con 0x7f100007bde0 2023-07-20T01:36:12.979 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.236+0000 7f100529a700 1 --2- 172.21.15.3:0/885086731 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0fec038630 0x7f0fec03aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:12.980 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.236+0000 7f1007cff700 1 -- 172.21.15.3:0/885086731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1000195fd0 con 0x7f100007bde0 2023-07-20T01:36:12.980 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.237+0000 7f100529a700 1 --2- 172.21.15.3:0/885086731 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0fec038630 0x7f0fec03aaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0006fd0 tx=0x7f0ff0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:12.980 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.246+0000 7f0ff6ffd700 1 -- 172.21.15.3:0/885086731 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ffc018db0 con 0x7f100007bde0 2023-07-20T01:36:12.981 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.631+0000 7f1007cff700 1 -- 172.21.15.3:0/885086731 --> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f100002cae0 con 0x7f0fec038630 2023-07-20T01:36:12.981 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.638+0000 7f0ff6ffd700 1 -- 172.21.15.3:0/885086731 <== mgr.14120 v2:172.21.15.3:6800/596448098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f100002cae0 con 0x7f0fec038630 2023-07-20T01:36:12.981 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.640+0000 7f1007cff700 1 -- 172.21.15.3:0/885086731 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0fec038630 msgr2=0x7f0fec03aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:12.981 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.640+0000 7f1007cff700 1 --2- 172.21.15.3:0/885086731 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0fec038630 0x7f0fec03aaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0006fd0 tx=0x7f0ff0006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:12.982 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.640+0000 7f1007cff700 1 -- 172.21.15.3:0/885086731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 msgr2=0x7f100019c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:12.982 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.640+0000 7f1007cff700 1 --2- 172.21.15.3:0/885086731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100019c230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009130 tx=0x7f0ffc011ad0 comp rx=0 tx=0).stop 2023-07-20T01:36:12.982 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.640+0000 7f1007cff700 1 -- 172.21.15.3:0/885086731 shutdown_connections 2023-07-20T01:36:12.983 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.640+0000 7f1007cff700 1 --2- 172.21.15.3:0/885086731 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0fec038630 0x7f0fec03aaf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:12.983 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.640+0000 7f1007cff700 1 --2- 172.21.15.3:0/885086731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f100007bde0 0x7f100019c230 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:12.983 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.640+0000 7f1007cff700 1 -- 172.21.15.3:0/885086731 >> 172.21.15.3:0/885086731 conn(0x7f10001015a0 msgr2=0x7f1000103910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:12.983 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.641+0000 7f1007cff700 1 -- 172.21.15.3:0/885086731 shutdown_connections 2023-07-20T01:36:12.984 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:12.641+0000 7f1007cff700 1 -- 172.21.15.3:0/885086731 wait complete. 2023-07-20T01:36:13.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:13 smithi003 ceph-mon[96422]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:36:13.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:13 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:13.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:13 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:36:13.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:13 smithi003 ceph-mon[96422]: mgrmap e8: x(active, since 2s) 2023-07-20T01:36:14.218 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-20T01:36:14.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.495+0000 7f616ee8f700 1 Processor -- start 2023-07-20T01:36:14.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.495+0000 7f616ee8f700 1 -- start start 2023-07-20T01:36:14.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.496+0000 7f616ee8f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f6168108e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:14.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.496+0000 7f616ee8f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6168073f70 con 0x7f6168106a80 2023-07-20T01:36:14.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.496+0000 7f616cc2b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f6168108e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:14.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.496+0000 7f616cc2b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f6168108e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41582/0 (socket says 172.21.15.3:41582) 2023-07-20T01:36:14.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.496+0000 7f616cc2b700 1 -- 172.21.15.3:0/489387538 learned_addr learned my addr 172.21.15.3:0/489387538 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:14.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.496+0000 7f616cc2b700 1 -- 172.21.15.3:0/489387538 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61681093b0 con 0x7f6168106a80 2023-07-20T01:36:14.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.496+0000 7f616cc2b700 1 --2- 172.21.15.3:0/489387538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f6168108e70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f615c0098a0 tx=0x7f615c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3d6fda8fffe4b9 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:14.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.496+0000 7f61677fe700 1 -- 172.21.15.3:0/489387538 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f615c004160 con 0x7f6168106a80 2023-07-20T01:36:14.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.497+0000 7f61677fe700 1 -- 172.21.15.3:0/489387538 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f615c0042c0 con 0x7f6168106a80 2023-07-20T01:36:14.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.497+0000 7f61677fe700 1 -- 172.21.15.3:0/489387538 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f615c011a00 con 0x7f6168106a80 2023-07-20T01:36:14.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.497+0000 7f616ee8f700 1 -- 172.21.15.3:0/489387538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 msgr2=0x7f6168108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:14.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.497+0000 7f616ee8f700 1 --2- 172.21.15.3:0/489387538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f6168108e70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f615c0098a0 tx=0x7f615c009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:14.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.497+0000 7f616ee8f700 1 -- 172.21.15.3:0/489387538 shutdown_connections 2023-07-20T01:36:14.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.497+0000 7f616ee8f700 1 --2- 172.21.15.3:0/489387538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f6168108e70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:14.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.497+0000 7f616ee8f700 1 -- 172.21.15.3:0/489387538 >> 172.21.15.3:0/489387538 conn(0x7f6168100490 msgr2=0x7f61681028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:14.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.497+0000 7f616ee8f700 1 -- 172.21.15.3:0/489387538 shutdown_connections 2023-07-20T01:36:14.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.497+0000 7f616ee8f700 1 -- 172.21.15.3:0/489387538 wait complete. 2023-07-20T01:36:14.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.498+0000 7f616ee8f700 1 Processor -- start 2023-07-20T01:36:14.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.498+0000 7f616ee8f700 1 -- start start 2023-07-20T01:36:14.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.498+0000 7f616ee8f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f61681063e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:14.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.498+0000 7f616ee8f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6168106920 con 0x7f6168106a80 2023-07-20T01:36:14.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.498+0000 7f616cc2b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f61681063e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:14.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.498+0000 7f616cc2b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f61681063e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41584/0 (socket says 172.21.15.3:41584) 2023-07-20T01:36:14.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.498+0000 7f616cc2b700 1 -- 172.21.15.3:0/1134464439 learned_addr learned my addr 172.21.15.3:0/1134464439 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:14.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.499+0000 7f616cc2b700 1 -- 172.21.15.3:0/1134464439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615c009550 con 0x7f6168106a80 2023-07-20T01:36:14.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.499+0000 7f616cc2b700 1 --2- 172.21.15.3:0/1134464439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f61681063e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f615c006af0 tx=0x7f615c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:14.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.499+0000 7f6165ffb700 1 -- 172.21.15.3:0/1134464439 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f615c003a40 con 0x7f6168106a80 2023-07-20T01:36:14.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.499+0000 7f6165ffb700 1 -- 172.21.15.3:0/1134464439 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f615c003ba0 con 0x7f6168106a80 2023-07-20T01:36:14.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.499+0000 7f616ee8f700 1 -- 172.21.15.3:0/1134464439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6168104a30 con 0x7f6168106a80 2023-07-20T01:36:14.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.499+0000 7f6165ffb700 1 -- 172.21.15.3:0/1134464439 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f615c01a6e0 con 0x7f6168106a80 2023-07-20T01:36:14.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.499+0000 7f616ee8f700 1 -- 172.21.15.3:0/1134464439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6168104e50 con 0x7f6168106a80 2023-07-20T01:36:14.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.500+0000 7f6165ffb700 1 -- 172.21.15.3:0/1134464439 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f615c024430 con 0x7f6168106a80 2023-07-20T01:36:14.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.500+0000 7f616ee8f700 1 -- 172.21.15.3:0/1134464439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616810a150 con 0x7f6168106a80 2023-07-20T01:36:14.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.500+0000 7f6165ffb700 1 --2- 172.21.15.3:0/1134464439 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6158038760 0x7f615803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:14.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.501+0000 7f6165ffb700 1 -- 172.21.15.3:0/1134464439 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f615c04c7b0 con 0x7f6168106a80 2023-07-20T01:36:14.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.501+0000 7f6167fff700 1 --2- 172.21.15.3:0/1134464439 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6158038760 0x7f615803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:14.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.501+0000 7f6167fff700 1 --2- 172.21.15.3:0/1134464439 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6158038760 0x7f615803ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6154006fd0 tx=0x7f6154006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:14.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.506+0000 7f6165ffb700 1 -- 172.21.15.3:0/1134464439 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f615c0246e0 con 0x7f6168106a80 2023-07-20T01:36:14.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.862+0000 7f616ee8f700 1 -- 172.21.15.3:0/1134464439 --> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f6168102790 con 0x7f6158038760 2023-07-20T01:36:14.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.863+0000 7f6165ffb700 1 -- 172.21.15.3:0/1134464439 <== mgr.14120 v2:172.21.15.3:6800/596448098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f6168102790 con 0x7f6158038760 2023-07-20T01:36:14.224 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.866+0000 7f616ee8f700 1 -- 172.21.15.3:0/1134464439 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6158038760 msgr2=0x7f615803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:14.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.866+0000 7f616ee8f700 1 --2- 172.21.15.3:0/1134464439 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6158038760 0x7f615803ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6154006fd0 tx=0x7f6154006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:14.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.866+0000 7f616ee8f700 1 -- 172.21.15.3:0/1134464439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 msgr2=0x7f61681063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:14.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.866+0000 7f616ee8f700 1 --2- 172.21.15.3:0/1134464439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f61681063e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f615c006af0 tx=0x7f615c011ad0 comp rx=0 tx=0).stop 2023-07-20T01:36:14.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.866+0000 7f616ee8f700 1 -- 172.21.15.3:0/1134464439 shutdown_connections 2023-07-20T01:36:14.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.866+0000 7f616ee8f700 1 --2- 172.21.15.3:0/1134464439 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6158038760 0x7f615803ac20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:14.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.866+0000 7f616ee8f700 1 --2- 172.21.15.3:0/1134464439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6168106a80 0x7f61681063e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:14.225 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.866+0000 7f616ee8f700 1 -- 172.21.15.3:0/1134464439 >> 172.21.15.3:0/1134464439 conn(0x7f6168100490 msgr2=0x7f6168101b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:14.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.866+0000 7f616ee8f700 1 -- 172.21.15.3:0/1134464439 shutdown_connections 2023-07-20T01:36:14.226 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:13.866+0000 7f616ee8f700 1 -- 172.21.15.3:0/1134464439 wait complete. 2023-07-20T01:36:14.226 INFO:teuthology.orchestra.run.smithi003.stdout:Generating ssh key... 2023-07-20T01:36:14.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:14 smithi003 ceph-mon[96422]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:36:15.608 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: Generating public/private rsa key pair. 2023-07-20T01:36:15.609 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: Your identification has been saved in /tmp/tmp9rcbrley/key. 2023-07-20T01:36:15.609 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: Your public key has been saved in /tmp/tmp9rcbrley/key.pub. 2023-07-20T01:36:15.609 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: The key fingerprint is: 2023-07-20T01:36:15.609 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: SHA256:U/GYuhBa3h3Fgued22qT+7jV6l6bjzqhXncNcEdf7Ts ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:36:15.609 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: The key's randomart image is: 2023-07-20T01:36:15.609 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: +---[RSA 3072]----+ 2023-07-20T01:36:15.610 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: | .... +| 2023-07-20T01:36:15.610 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: | . o*. .+| 2023-07-20T01:36:15.610 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: | o o=oo...o| 2023-07-20T01:36:15.610 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: | + o +..oo ..| 2023-07-20T01:36:15.610 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: | . o S . o. .| 2023-07-20T01:36:15.610 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: | . o o .E.| 2023-07-20T01:36:15.610 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: | . ..+o *| 2023-07-20T01:36:15.611 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: | ..*+ =o| 2023-07-20T01:36:15.611 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: | ...=X*oo| 2023-07-20T01:36:15.611 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: +----[SHA256]-----+ 2023-07-20T01:36:15.671 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.733+0000 7fdceb489700 1 Processor -- start 2023-07-20T01:36:15.671 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.733+0000 7fdceb489700 1 -- start start 2023-07-20T01:36:15.671 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.733+0000 7fdceb489700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce4108e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:15.671 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.733+0000 7fdceb489700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce4073f70 con 0x7fdce4106a80 2023-07-20T01:36:15.671 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.733+0000 7fdce9225700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce4108e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:15.672 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.733+0000 7fdce9225700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce4108e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41588/0 (socket says 172.21.15.3:41588) 2023-07-20T01:36:15.672 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.733+0000 7fdce9225700 1 -- 172.21.15.3:0/3037938646 learned_addr learned my addr 172.21.15.3:0/3037938646 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:15.672 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.734+0000 7fdce9225700 1 -- 172.21.15.3:0/3037938646 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce41093b0 con 0x7fdce4106a80 2023-07-20T01:36:15.672 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.734+0000 7fdce9225700 1 --2- 172.21.15.3:0/3037938646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce4108e70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdce00098a0 tx=0x7fdce0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7749aa592ef248 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:15.672 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.734+0000 7fdcdbfff700 1 -- 172.21.15.3:0/3037938646 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdce0004160 con 0x7fdce4106a80 2023-07-20T01:36:15.672 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.734+0000 7fdcdbfff700 1 -- 172.21.15.3:0/3037938646 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdce00042c0 con 0x7fdce4106a80 2023-07-20T01:36:15.672 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.734+0000 7fdcdbfff700 1 -- 172.21.15.3:0/3037938646 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdce0011a00 con 0x7fdce4106a80 2023-07-20T01:36:15.673 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.735+0000 7fdceb489700 1 -- 172.21.15.3:0/3037938646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 msgr2=0x7fdce4108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:15.673 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.735+0000 7fdceb489700 1 --2- 172.21.15.3:0/3037938646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce4108e70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdce00098a0 tx=0x7fdce0009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:15.673 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.735+0000 7fdceb489700 1 -- 172.21.15.3:0/3037938646 shutdown_connections 2023-07-20T01:36:15.673 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.735+0000 7fdceb489700 1 --2- 172.21.15.3:0/3037938646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce4108e70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:15.673 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.735+0000 7fdceb489700 1 -- 172.21.15.3:0/3037938646 >> 172.21.15.3:0/3037938646 conn(0x7fdce4100490 msgr2=0x7fdce41028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:15.673 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.735+0000 7fdceb489700 1 -- 172.21.15.3:0/3037938646 shutdown_connections 2023-07-20T01:36:15.673 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.735+0000 7fdceb489700 1 -- 172.21.15.3:0/3037938646 wait complete. 2023-07-20T01:36:15.673 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.736+0000 7fdceb489700 1 Processor -- start 2023-07-20T01:36:15.674 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.736+0000 7fdceb489700 1 -- start start 2023-07-20T01:36:15.674 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.736+0000 7fdceb489700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce41063e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:15.674 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.736+0000 7fdceb489700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce4106920 con 0x7fdce4106a80 2023-07-20T01:36:15.674 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.736+0000 7fdce9225700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce41063e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:15.674 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.736+0000 7fdce9225700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce41063e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41590/0 (socket says 172.21.15.3:41590) 2023-07-20T01:36:15.674 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.736+0000 7fdce9225700 1 -- 172.21.15.3:0/328043212 learned_addr learned my addr 172.21.15.3:0/328043212 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:15.674 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.736+0000 7fdce9225700 1 -- 172.21.15.3:0/328043212 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce0009550 con 0x7fdce4106a80 2023-07-20T01:36:15.674 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.737+0000 7fdce9225700 1 --2- 172.21.15.3:0/328043212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce41063e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdce0006af0 tx=0x7fdce0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:15.675 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.737+0000 7fdcda7fc700 1 -- 172.21.15.3:0/328043212 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdce0003a40 con 0x7fdce4106a80 2023-07-20T01:36:15.675 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.737+0000 7fdceb489700 1 -- 172.21.15.3:0/328043212 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce4104a30 con 0x7fdce4106a80 2023-07-20T01:36:15.675 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.737+0000 7fdcda7fc700 1 -- 172.21.15.3:0/328043212 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdce0003ba0 con 0x7fdce4106a80 2023-07-20T01:36:15.675 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.737+0000 7fdcda7fc700 1 -- 172.21.15.3:0/328043212 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdce001a6e0 con 0x7fdce4106a80 2023-07-20T01:36:15.675 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.737+0000 7fdceb489700 1 -- 172.21.15.3:0/328043212 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce4104e50 con 0x7fdce4106a80 2023-07-20T01:36:15.675 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.738+0000 7fdcda7fc700 1 -- 172.21.15.3:0/328043212 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fdce001a950 con 0x7fdce4106a80 2023-07-20T01:36:15.675 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.738+0000 7fdcda7fc700 1 --2- 172.21.15.3:0/328043212 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fdcd0038710 0x7fdcd003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:15.675 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.738+0000 7fdcda7fc700 1 -- 172.21.15.3:0/328043212 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdce004c7b0 con 0x7fdce4106a80 2023-07-20T01:36:15.676 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.738+0000 7fdce8a24700 1 --2- 172.21.15.3:0/328043212 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fdcd0038710 0x7fdcd003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:15.676 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.738+0000 7fdceb489700 1 -- 172.21.15.3:0/328043212 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce410a150 con 0x7fdce4106a80 2023-07-20T01:36:15.676 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.738+0000 7fdce8a24700 1 --2- 172.21.15.3:0/328043212 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fdcd0038710 0x7fdcd003abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4006fd0 tx=0x7fdcd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:15.676 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:14.743+0000 7fdcda7fc700 1 -- 172.21.15.3:0/328043212 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdce001ac00 con 0x7fdce4106a80 2023-07-20T01:36:15.676 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.100+0000 7fdceb489700 1 -- 172.21.15.3:0/328043212 --> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fdce4102790 con 0x7fdcd0038710 2023-07-20T01:36:15.676 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.319+0000 7fdcda7fc700 1 -- 172.21.15.3:0/328043212 <== mgr.14120 v2:172.21.15.3:6800/596448098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdce4102790 con 0x7fdcd0038710 2023-07-20T01:36:15.676 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.321+0000 7fdceb489700 1 -- 172.21.15.3:0/328043212 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fdcd0038710 msgr2=0x7fdcd003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:15.677 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.321+0000 7fdceb489700 1 --2- 172.21.15.3:0/328043212 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fdcd0038710 0x7fdcd003abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4006fd0 tx=0x7fdcd4006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:15.677 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.321+0000 7fdceb489700 1 -- 172.21.15.3:0/328043212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 msgr2=0x7fdce41063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:15.677 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.321+0000 7fdceb489700 1 --2- 172.21.15.3:0/328043212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce41063e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdce0006af0 tx=0x7fdce0011ad0 comp rx=0 tx=0).stop 2023-07-20T01:36:15.677 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.321+0000 7fdceb489700 1 -- 172.21.15.3:0/328043212 shutdown_connections 2023-07-20T01:36:15.677 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.321+0000 7fdceb489700 1 --2- 172.21.15.3:0/328043212 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fdcd0038710 0x7fdcd003abd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:15.677 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.321+0000 7fdceb489700 1 --2- 172.21.15.3:0/328043212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdce4106a80 0x7fdce41063e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:15.677 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.321+0000 7fdceb489700 1 -- 172.21.15.3:0/328043212 >> 172.21.15.3:0/328043212 conn(0x7fdce4100490 msgr2=0x7fdce4101b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:15.677 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.321+0000 7fdceb489700 1 -- 172.21.15.3:0/328043212 shutdown_connections 2023-07-20T01:36:15.678 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:15.321+0000 7fdceb489700 1 -- 172.21.15.3:0/328043212 wait complete. 2023-07-20T01:36:15.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-mon[96422]: [20/Jul/2023:01:36:14] ENGINE Bus STARTING 2023-07-20T01:36:15.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-mon[96422]: [20/Jul/2023:01:36:14] ENGINE Serving on http://172.21.15.3:8765 2023-07-20T01:36:15.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-mon[96422]: [20/Jul/2023:01:36:15] ENGINE Serving on https://172.21.15.3:7150 2023-07-20T01:36:15.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-mon[96422]: [20/Jul/2023:01:36:15] ENGINE Bus STARTED 2023-07-20T01:36:15.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-mon[96422]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:36:15.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-mon[96422]: Generating ssh key... 2023-07-20T01:36:15.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:36:15.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:15.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:15 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:16.915 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdq/XYHDg/SpMUlL9fgndxVP3LLufcPhBfdORMhq0puHfPWSwQTzoOnbnyUdZQOiXssoMIprxef9rwd1SPLxtZOwxqos4chnOvcwoPg228YRjh4p6xAOY8X9bHJFcNVX+gvPZGYFipnjO6Svl/i2PJwi4AwG9wUFmTIYkFq+dOA0t/x/zMKqTVCj5jziEkQrZp73bad14r3kFjkFBPMyjbB+rGiUKfTRcv2J6EjVyxjMnwlknEXGy3pdbKhGLSBz3FpneAw9nPFZX+9YJC8x4k/U1BmFmWhX7B0KxmYhN9r6ei3jFhCx2ytcDrin76Inxz/8iMAEMR5BoHy5//ajElTsSUDyhHZDyQ4oTrmrG8z0J97GPz8DzMC4LAzgrwtYpnFDYhtP3kn5jHPGOL6X8DuY2oKr8AISIUvznFy8t2XudOvh2DB/tFbURFu8DWx1wWGZnKOIqid6qp3prJWvOPGIOVwsxPiFe36ZQdRZHX5yh0t4dDKbMiOV0rzvu861U= ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:36:16.916 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.220+0000 7f0d500cd700 1 Processor -- start 2023-07-20T01:36:16.916 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.220+0000 7f0d500cd700 1 -- start start 2023-07-20T01:36:16.916 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.221+0000 7f0d500cd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48106920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:16.916 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.221+0000 7f0d500cd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d48106ef0 con 0x7f0d48106500 2023-07-20T01:36:16.917 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.221+0000 7f0d4de69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48106920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:16.917 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.221+0000 7f0d4de69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48106920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41602/0 (socket says 172.21.15.3:41602) 2023-07-20T01:36:16.917 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.221+0000 7f0d4de69700 1 -- 172.21.15.3:0/115733335 learned_addr learned my addr 172.21.15.3:0/115733335 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:16.917 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.221+0000 7f0d4de69700 1 -- 172.21.15.3:0/115733335 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d48105160 con 0x7f0d48106500 2023-07-20T01:36:16.917 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.221+0000 7f0d4de69700 1 --2- 172.21.15.3:0/115733335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48106920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0098a0 tx=0x7f0d3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504d8ac6646f09d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:16.917 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.221+0000 7f0d4ce67700 1 -- 172.21.15.3:0/115733335 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c004160 con 0x7f0d48106500 2023-07-20T01:36:16.917 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.221+0000 7f0d4ce67700 1 -- 172.21.15.3:0/115733335 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d3c0042c0 con 0x7f0d48106500 2023-07-20T01:36:16.918 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.221+0000 7f0d4ce67700 1 -- 172.21.15.3:0/115733335 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c011a00 con 0x7f0d48106500 2023-07-20T01:36:16.918 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.222+0000 7f0d500cd700 1 -- 172.21.15.3:0/115733335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 msgr2=0x7f0d48106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:16.918 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.222+0000 7f0d500cd700 1 --2- 172.21.15.3:0/115733335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48106920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0098a0 tx=0x7f0d3c009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:16.918 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.222+0000 7f0d500cd700 1 -- 172.21.15.3:0/115733335 shutdown_connections 2023-07-20T01:36:16.918 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.222+0000 7f0d500cd700 1 --2- 172.21.15.3:0/115733335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48106920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:16.918 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.222+0000 7f0d500cd700 1 -- 172.21.15.3:0/115733335 >> 172.21.15.3:0/115733335 conn(0x7f0d480ffc10 msgr2=0x7f0d48102050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:16.918 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.222+0000 7f0d500cd700 1 -- 172.21.15.3:0/115733335 shutdown_connections 2023-07-20T01:36:16.919 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.222+0000 7f0d500cd700 1 -- 172.21.15.3:0/115733335 wait complete. 2023-07-20T01:36:16.919 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.223+0000 7f0d500cd700 1 Processor -- start 2023-07-20T01:36:16.919 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.223+0000 7f0d500cd700 1 -- start start 2023-07-20T01:36:16.919 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.223+0000 7f0d500cd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48190090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:16.919 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.223+0000 7f0d500cd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d481905d0 con 0x7f0d48106500 2023-07-20T01:36:16.919 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.223+0000 7f0d4de69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48190090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:16.919 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.223+0000 7f0d4de69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48190090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41604/0 (socket says 172.21.15.3:41604) 2023-07-20T01:36:16.920 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.223+0000 7f0d4de69700 1 -- 172.21.15.3:0/2321285552 learned_addr learned my addr 172.21.15.3:0/2321285552 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:16.920 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.223+0000 7f0d4de69700 1 -- 172.21.15.3:0/2321285552 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c009550 con 0x7f0d48106500 2023-07-20T01:36:16.920 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.223+0000 7f0d4de69700 1 --2- 172.21.15.3:0/2321285552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48190090 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c009130 tx=0x7f0d3c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:16.920 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.224+0000 7f0d3affd700 1 -- 172.21.15.3:0/2321285552 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c003ab0 con 0x7f0d48106500 2023-07-20T01:36:16.920 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.224+0000 7f0d3affd700 1 -- 172.21.15.3:0/2321285552 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d3c003c10 con 0x7f0d48106500 2023-07-20T01:36:16.920 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.224+0000 7f0d500cd700 1 -- 172.21.15.3:0/2321285552 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4818e6e0 con 0x7f0d48106500 2023-07-20T01:36:16.920 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.224+0000 7f0d3affd700 1 -- 172.21.15.3:0/2321285552 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c0247d0 con 0x7f0d48106500 2023-07-20T01:36:16.921 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.224+0000 7f0d500cd700 1 -- 172.21.15.3:0/2321285552 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4818eb00 con 0x7f0d48106500 2023-07-20T01:36:16.921 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.225+0000 7f0d3affd700 1 -- 172.21.15.3:0/2321285552 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f0d3c01a430 con 0x7f0d48106500 2023-07-20T01:36:16.921 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.225+0000 7f0d3affd700 1 --2- 172.21.15.3:0/2321285552 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0d34038760 0x7f0d3403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:16.921 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.225+0000 7f0d500cd700 1 -- 172.21.15.3:0/2321285552 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d48061da0 con 0x7f0d48106500 2023-07-20T01:36:16.921 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.225+0000 7f0d3affd700 1 -- 172.21.15.3:0/2321285552 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0d3c04c7d0 con 0x7f0d48106500 2023-07-20T01:36:16.921 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.225+0000 7f0d4d668700 1 --2- 172.21.15.3:0/2321285552 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0d34038760 0x7f0d3403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:16.922 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.225+0000 7f0d4d668700 1 --2- 172.21.15.3:0/2321285552 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0d34038760 0x7f0d3403ac20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d44006fd0 tx=0x7f0d44006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:16.922 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.230+0000 7f0d3affd700 1 -- 172.21.15.3:0/2321285552 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0d3c01a730 con 0x7f0d48106500 2023-07-20T01:36:16.922 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.592+0000 7f0d500cd700 1 -- 172.21.15.3:0/2321285552 --> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d48101e70 con 0x7f0d34038760 2023-07-20T01:36:16.922 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.593+0000 7f0d3affd700 1 -- 172.21.15.3:0/2321285552 <== mgr.14120 v2:172.21.15.3:6800/596448098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f0d48101e70 con 0x7f0d34038760 2023-07-20T01:36:16.922 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.595+0000 7f0d500cd700 1 -- 172.21.15.3:0/2321285552 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0d34038760 msgr2=0x7f0d3403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:16.922 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.595+0000 7f0d500cd700 1 --2- 172.21.15.3:0/2321285552 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0d34038760 0x7f0d3403ac20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d44006fd0 tx=0x7f0d44006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:16.922 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.595+0000 7f0d500cd700 1 -- 172.21.15.3:0/2321285552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 msgr2=0x7f0d48190090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:16.923 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.595+0000 7f0d500cd700 1 --2- 172.21.15.3:0/2321285552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48190090 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c009130 tx=0x7f0d3c011ad0 comp rx=0 tx=0).stop 2023-07-20T01:36:16.923 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.596+0000 7f0d500cd700 1 -- 172.21.15.3:0/2321285552 shutdown_connections 2023-07-20T01:36:16.923 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.596+0000 7f0d500cd700 1 --2- 172.21.15.3:0/2321285552 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f0d34038760 0x7f0d3403ac20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:16.923 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.596+0000 7f0d500cd700 1 --2- 172.21.15.3:0/2321285552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d48106500 0x7f0d48190090 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:16.923 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.596+0000 7f0d500cd700 1 -- 172.21.15.3:0/2321285552 >> 172.21.15.3:0/2321285552 conn(0x7f0d480ffc10 msgr2=0x7f0d481049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:16.923 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.596+0000 7f0d500cd700 1 -- 172.21.15.3:0/2321285552 shutdown_connections 2023-07-20T01:36:16.923 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:16.596+0000 7f0d500cd700 1 -- 172.21.15.3:0/2321285552 wait complete. 2023-07-20T01:36:16.924 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-20T01:36:16.924 INFO:teuthology.orchestra.run.smithi003.stdout:Adding key to root@localhost authorized_keys... 2023-07-20T01:36:16.924 INFO:teuthology.orchestra.run.smithi003.stdout:Adding host smithi003... 2023-07-20T01:36:18.331 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:18 smithi003 ceph-mon[96422]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:36:19.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:19 smithi003 ceph-mon[96422]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:36:19.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:19 smithi003 ceph-mon[96422]: Deploying cephadm binary to smithi003 2023-07-20T01:36:20.868 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Added host 'smithi003' with addr '172.21.15.3' 2023-07-20T01:36:20.868 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.448+0000 7eff76f82700 1 Processor -- start 2023-07-20T01:36:20.868 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.448+0000 7eff76f82700 1 -- start start 2023-07-20T01:36:20.868 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.448+0000 7eff76f82700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70108e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:20.869 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.448+0000 7eff76f82700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff70073f50 con 0x7eff70106aa0 2023-07-20T01:36:20.869 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.448+0000 7eff74d1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70108e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:20.869 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.448+0000 7eff74d1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70108e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41608/0 (socket says 172.21.15.3:41608) 2023-07-20T01:36:20.869 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.448+0000 7eff74d1e700 1 -- 172.21.15.3:0/198474399 learned_addr learned my addr 172.21.15.3:0/198474399 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:20.869 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.449+0000 7eff74d1e700 1 -- 172.21.15.3:0/198474399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff701093d0 con 0x7eff70106aa0 2023-07-20T01:36:20.869 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.449+0000 7eff74d1e700 1 --2- 172.21.15.3:0/198474399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70108e90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7eff640098a0 tx=0x7eff64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b4e9793dc4a5382 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:20.869 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.449+0000 7eff6f7fe700 1 -- 172.21.15.3:0/198474399 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff64004160 con 0x7eff70106aa0 2023-07-20T01:36:20.870 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.449+0000 7eff6f7fe700 1 -- 172.21.15.3:0/198474399 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff640042c0 con 0x7eff70106aa0 2023-07-20T01:36:20.870 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.449+0000 7eff6f7fe700 1 -- 172.21.15.3:0/198474399 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff64011a00 con 0x7eff70106aa0 2023-07-20T01:36:20.870 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.450+0000 7eff76f82700 1 -- 172.21.15.3:0/198474399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 msgr2=0x7eff70108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:20.870 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.450+0000 7eff76f82700 1 --2- 172.21.15.3:0/198474399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70108e90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7eff640098a0 tx=0x7eff64009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:20.870 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.450+0000 7eff76f82700 1 -- 172.21.15.3:0/198474399 shutdown_connections 2023-07-20T01:36:20.870 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.450+0000 7eff76f82700 1 --2- 172.21.15.3:0/198474399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70108e90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:20.870 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.450+0000 7eff76f82700 1 -- 172.21.15.3:0/198474399 >> 172.21.15.3:0/198474399 conn(0x7eff70100490 msgr2=0x7eff701028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:20.870 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.450+0000 7eff76f82700 1 -- 172.21.15.3:0/198474399 shutdown_connections 2023-07-20T01:36:20.871 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.450+0000 7eff76f82700 1 -- 172.21.15.3:0/198474399 wait complete. 2023-07-20T01:36:20.871 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.450+0000 7eff76f82700 1 Processor -- start 2023-07-20T01:36:20.871 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.450+0000 7eff76f82700 1 -- start start 2023-07-20T01:36:20.871 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.451+0000 7eff76f82700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70106400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:20.871 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.451+0000 7eff76f82700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff70106940 con 0x7eff70106aa0 2023-07-20T01:36:20.871 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.451+0000 7eff74d1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70106400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:20.871 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.451+0000 7eff74d1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70106400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41610/0 (socket says 172.21.15.3:41610) 2023-07-20T01:36:20.871 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.451+0000 7eff74d1e700 1 -- 172.21.15.3:0/1343006363 learned_addr learned my addr 172.21.15.3:0/1343006363 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:20.872 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.451+0000 7eff74d1e700 1 -- 172.21.15.3:0/1343006363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff64009550 con 0x7eff70106aa0 2023-07-20T01:36:20.872 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.451+0000 7eff74d1e700 1 --2- 172.21.15.3:0/1343006363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70106400 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7eff64006af0 tx=0x7eff64011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:20.872 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.451+0000 7eff6dffb700 1 -- 172.21.15.3:0/1343006363 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff64003a40 con 0x7eff70106aa0 2023-07-20T01:36:20.872 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.451+0000 7eff6dffb700 1 -- 172.21.15.3:0/1343006363 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff64003ba0 con 0x7eff70106aa0 2023-07-20T01:36:20.872 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.451+0000 7eff6dffb700 1 -- 172.21.15.3:0/1343006363 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff6401a6e0 con 0x7eff70106aa0 2023-07-20T01:36:20.872 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.451+0000 7eff76f82700 1 -- 172.21.15.3:0/1343006363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff70104a50 con 0x7eff70106aa0 2023-07-20T01:36:20.872 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.452+0000 7eff76f82700 1 -- 172.21.15.3:0/1343006363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff70104e70 con 0x7eff70106aa0 2023-07-20T01:36:20.873 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.453+0000 7eff6dffb700 1 -- 172.21.15.3:0/1343006363 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7eff64024a60 con 0x7eff70106aa0 2023-07-20T01:36:20.873 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.453+0000 7eff6dffb700 1 --2- 172.21.15.3:0/1343006363 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7eff60038760 0x7eff6003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:20.873 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.453+0000 7eff76f82700 1 -- 172.21.15.3:0/1343006363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7010a170 con 0x7eff70106aa0 2023-07-20T01:36:20.873 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.453+0000 7eff6dffb700 1 -- 172.21.15.3:0/1343006363 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7eff6404c700 con 0x7eff70106aa0 2023-07-20T01:36:20.873 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.453+0000 7eff6ffff700 1 --2- 172.21.15.3:0/1343006363 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7eff60038760 0x7eff6003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:20.873 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.453+0000 7eff6ffff700 1 --2- 172.21.15.3:0/1343006363 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7eff60038760 0x7eff6003ac20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7eff5c006fd0 tx=0x7eff5c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:20.873 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.458+0000 7eff6dffb700 1 -- 172.21.15.3:0/1343006363 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7eff64024430 con 0x7eff70106aa0 2023-07-20T01:36:20.873 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:17.822+0000 7eff76f82700 1 -- 172.21.15.3:0/1343006363 --> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}) v1 -- 0x7eff701027b0 con 0x7eff60038760 2023-07-20T01:36:20.874 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:20.567+0000 7eff6dffb700 1 -- 172.21.15.3:0/1343006363 <== mgr.14120 v2:172.21.15.3:6800/596448098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7eff701027b0 con 0x7eff60038760 2023-07-20T01:36:20.874 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:20.570+0000 7eff76f82700 1 -- 172.21.15.3:0/1343006363 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7eff60038760 msgr2=0x7eff6003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:20.874 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:20.570+0000 7eff76f82700 1 --2- 172.21.15.3:0/1343006363 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7eff60038760 0x7eff6003ac20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7eff5c006fd0 tx=0x7eff5c006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:20.874 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:20.570+0000 7eff76f82700 1 -- 172.21.15.3:0/1343006363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 msgr2=0x7eff70106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:20.874 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:20.570+0000 7eff76f82700 1 --2- 172.21.15.3:0/1343006363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70106400 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7eff64006af0 tx=0x7eff64011ad0 comp rx=0 tx=0).stop 2023-07-20T01:36:20.874 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:20.570+0000 7eff76f82700 1 -- 172.21.15.3:0/1343006363 shutdown_connections 2023-07-20T01:36:20.874 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:20.570+0000 7eff76f82700 1 --2- 172.21.15.3:0/1343006363 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7eff60038760 0x7eff6003ac20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:20.874 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:20.570+0000 7eff76f82700 1 --2- 172.21.15.3:0/1343006363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff70106aa0 0x7eff70106400 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:20.875 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:20.570+0000 7eff76f82700 1 -- 172.21.15.3:0/1343006363 >> 172.21.15.3:0/1343006363 conn(0x7eff70100490 msgr2=0x7eff70101b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:20.875 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:20.570+0000 7eff76f82700 1 -- 172.21.15.3:0/1343006363 shutdown_connections 2023-07-20T01:36:20.875 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:20.570+0000 7eff76f82700 1 -- 172.21.15.3:0/1343006363 wait complete. 2023-07-20T01:36:20.875 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mon service... 2023-07-20T01:36:21.798 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:21 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:21.799 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:21 smithi003 ceph-mon[96422]: Added host smithi003 2023-07-20T01:36:21.799 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:21 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:36:22.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-20T01:36:22.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.387+0000 7f12c98e0700 1 Processor -- start 2023-07-20T01:36:22.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.387+0000 7f12c98e0700 1 -- start start 2023-07-20T01:36:22.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.387+0000 7f12c98e0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c41046f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:22.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.387+0000 7f12c98e0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c4104cc0 con 0x7f12c41042d0 2023-07-20T01:36:22.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.387+0000 7f12c2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c41046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:22.213 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.387+0000 7f12c2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c41046f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41644/0 (socket says 172.21.15.3:41644) 2023-07-20T01:36:22.214 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.387+0000 7f12c2ffd700 1 -- 172.21.15.3:0/3651494595 learned_addr learned my addr 172.21.15.3:0/3651494595 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:22.214 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.387+0000 7f12c2ffd700 1 -- 172.21.15.3:0/3651494595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c4107260 con 0x7f12c41042d0 2023-07-20T01:36:22.214 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.387+0000 7f12c2ffd700 1 --2- 172.21.15.3:0/3651494595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c41046f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f12b40098a0 tx=0x7f12b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4f19c3930d2ec9 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:22.214 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.388+0000 7f12c1ffb700 1 -- 172.21.15.3:0/3651494595 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12b4004160 con 0x7f12c41042d0 2023-07-20T01:36:22.214 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.388+0000 7f12c1ffb700 1 -- 172.21.15.3:0/3651494595 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12b40042c0 con 0x7f12c41042d0 2023-07-20T01:36:22.214 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.388+0000 7f12c1ffb700 1 -- 172.21.15.3:0/3651494595 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12b4011a00 con 0x7f12c41042d0 2023-07-20T01:36:22.215 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.388+0000 7f12c98e0700 1 -- 172.21.15.3:0/3651494595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 msgr2=0x7f12c41046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:22.215 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.388+0000 7f12c98e0700 1 --2- 172.21.15.3:0/3651494595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c41046f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f12b40098a0 tx=0x7f12b4009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:22.215 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.388+0000 7f12c98e0700 1 -- 172.21.15.3:0/3651494595 shutdown_connections 2023-07-20T01:36:22.215 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.388+0000 7f12c98e0700 1 --2- 172.21.15.3:0/3651494595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c41046f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:22.215 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.388+0000 7f12c98e0700 1 -- 172.21.15.3:0/3651494595 >> 172.21.15.3:0/3651494595 conn(0x7f12c40ffc20 msgr2=0x7f12c4102040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:22.215 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.388+0000 7f12c98e0700 1 -- 172.21.15.3:0/3651494595 shutdown_connections 2023-07-20T01:36:22.216 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.388+0000 7f12c98e0700 1 -- 172.21.15.3:0/3651494595 wait complete. 2023-07-20T01:36:22.216 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.389+0000 7f12c98e0700 1 Processor -- start 2023-07-20T01:36:22.216 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.389+0000 7f12c98e0700 1 -- start start 2023-07-20T01:36:22.216 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.389+0000 7f12c98e0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c4199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:22.216 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.389+0000 7f12c98e0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c419a250 con 0x7f12c41042d0 2023-07-20T01:36:22.216 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.389+0000 7f12c2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c4199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:22.217 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.389+0000 7f12c2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c4199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41646/0 (socket says 172.21.15.3:41646) 2023-07-20T01:36:22.217 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.389+0000 7f12c2ffd700 1 -- 172.21.15.3:0/3514869859 learned_addr learned my addr 172.21.15.3:0/3514869859 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:22.217 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.390+0000 7f12c2ffd700 1 -- 172.21.15.3:0/3514869859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4009550 con 0x7f12c41042d0 2023-07-20T01:36:22.217 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.390+0000 7f12c2ffd700 1 --2- 172.21.15.3:0/3514869859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c4199d10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009fb0 tx=0x7f12b4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:22.217 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.390+0000 7f12c88de700 1 -- 172.21.15.3:0/3514869859 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12b4003930 con 0x7f12c41042d0 2023-07-20T01:36:22.217 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.390+0000 7f12c88de700 1 -- 172.21.15.3:0/3514869859 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12b4003a90 con 0x7f12c41042d0 2023-07-20T01:36:22.217 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.390+0000 7f12c98e0700 1 -- 172.21.15.3:0/3514869859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c419a450 con 0x7f12c41042d0 2023-07-20T01:36:22.218 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.390+0000 7f12c88de700 1 -- 172.21.15.3:0/3514869859 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12b40195e0 con 0x7f12c41042d0 2023-07-20T01:36:22.218 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.390+0000 7f12c98e0700 1 -- 172.21.15.3:0/3514869859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c419a870 con 0x7f12c41042d0 2023-07-20T01:36:22.218 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.391+0000 7f12c88de700 1 -- 172.21.15.3:0/3514869859 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f12b40198a0 con 0x7f12c41042d0 2023-07-20T01:36:22.218 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.391+0000 7f12c88de700 1 --2- 172.21.15.3:0/3514869859 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f12ac0386c0 0x7f12ac03ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:22.218 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.391+0000 7f12c88de700 1 -- 172.21.15.3:0/3514869859 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f12b404c8d0 con 0x7f12c41042d0 2023-07-20T01:36:22.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.391+0000 7f12c27fc700 1 --2- 172.21.15.3:0/3514869859 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f12ac0386c0 0x7f12ac03ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:22.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.391+0000 7f12c98e0700 1 -- 172.21.15.3:0/3514869859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c4061da0 con 0x7f12c41042d0 2023-07-20T01:36:22.219 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.392+0000 7f12c27fc700 1 --2- 172.21.15.3:0/3514869859 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f12ac0386c0 0x7f12ac03ab80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f12b8006fd0 tx=0x7f12b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:22.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.396+0000 7f12c88de700 1 -- 172.21.15.3:0/3514869859 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f12b4019ba0 con 0x7f12c41042d0 2023-07-20T01:36:22.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.786+0000 7f12c98e0700 1 -- 172.21.15.3:0/3514869859 --> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f12c4101f30 con 0x7f12ac0386c0 2023-07-20T01:36:22.220 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.791+0000 7f12c88de700 1 -- 172.21.15.3:0/3514869859 <== mgr.14120 v2:172.21.15.3:6800/596448098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f12c4101f30 con 0x7f12ac0386c0 2023-07-20T01:36:22.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.793+0000 7f12c98e0700 1 -- 172.21.15.3:0/3514869859 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f12ac0386c0 msgr2=0x7f12ac03ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:22.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.793+0000 7f12c98e0700 1 --2- 172.21.15.3:0/3514869859 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f12ac0386c0 0x7f12ac03ab80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f12b8006fd0 tx=0x7f12b8006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:22.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.793+0000 7f12c98e0700 1 -- 172.21.15.3:0/3514869859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 msgr2=0x7f12c4199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:22.221 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.793+0000 7f12c98e0700 1 --2- 172.21.15.3:0/3514869859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c4199d10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009fb0 tx=0x7f12b4011ad0 comp rx=0 tx=0).stop 2023-07-20T01:36:22.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.793+0000 7f12c98e0700 1 -- 172.21.15.3:0/3514869859 shutdown_connections 2023-07-20T01:36:22.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.793+0000 7f12c98e0700 1 --2- 172.21.15.3:0/3514869859 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f12ac0386c0 0x7f12ac03ab80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:22.222 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.793+0000 7f12c98e0700 1 --2- 172.21.15.3:0/3514869859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12c41042d0 0x7f12c4199d10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:22.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.793+0000 7f12c98e0700 1 -- 172.21.15.3:0/3514869859 >> 172.21.15.3:0/3514869859 conn(0x7f12c40ffc20 msgr2=0x7f12c4101310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:22.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.793+0000 7f12c98e0700 1 -- 172.21.15.3:0/3514869859 shutdown_connections 2023-07-20T01:36:22.223 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:21.793+0000 7f12c98e0700 1 -- 172.21.15.3:0/3514869859 wait complete. 2023-07-20T01:36:22.224 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mgr service... 2023-07-20T01:36:23.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:22 smithi003 ceph-mon[96422]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:36:23.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:22 smithi003 ceph-mon[96422]: Saving service mon spec with placement count:5 2023-07-20T01:36:23.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:22 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:23.459 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-20T01:36:23.460 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.784+0000 7ffa5b7be700 1 Processor -- start 2023-07-20T01:36:23.460 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.784+0000 7ffa5b7be700 1 -- start start 2023-07-20T01:36:23.460 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.785+0000 7ffa5b7be700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa54079130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:23.460 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.785+0000 7ffa5b7be700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa54079700 con 0x7ffa5407acd0 2023-07-20T01:36:23.460 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.785+0000 7ffa5955a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa54079130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:23.460 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.785+0000 7ffa5955a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa54079130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41650/0 (socket says 172.21.15.3:41650) 2023-07-20T01:36:23.461 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.785+0000 7ffa5955a700 1 -- 172.21.15.3:0/3564591125 learned_addr learned my addr 172.21.15.3:0/3564591125 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:23.461 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.785+0000 7ffa5955a700 1 -- 172.21.15.3:0/3564591125 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa54079840 con 0x7ffa5407acd0 2023-07-20T01:36:23.461 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.785+0000 7ffa5955a700 1 --2- 172.21.15.3:0/3564591125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa54079130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffa500098a0 tx=0x7ffa50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91296fa7aa1f5d5c server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:23.461 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.785+0000 7ffa47fff700 1 -- 172.21.15.3:0/3564591125 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa50004160 con 0x7ffa5407acd0 2023-07-20T01:36:23.461 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.786+0000 7ffa47fff700 1 -- 172.21.15.3:0/3564591125 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa500042c0 con 0x7ffa5407acd0 2023-07-20T01:36:23.461 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.786+0000 7ffa47fff700 1 -- 172.21.15.3:0/3564591125 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa50011a00 con 0x7ffa5407acd0 2023-07-20T01:36:23.461 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.786+0000 7ffa5b7be700 1 -- 172.21.15.3:0/3564591125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 msgr2=0x7ffa54079130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:23.462 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.786+0000 7ffa5b7be700 1 --2- 172.21.15.3:0/3564591125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa54079130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffa500098a0 tx=0x7ffa50009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:23.462 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.786+0000 7ffa5b7be700 1 -- 172.21.15.3:0/3564591125 shutdown_connections 2023-07-20T01:36:23.462 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.786+0000 7ffa5b7be700 1 --2- 172.21.15.3:0/3564591125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa54079130 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:23.462 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.786+0000 7ffa5b7be700 1 -- 172.21.15.3:0/3564591125 >> 172.21.15.3:0/3564591125 conn(0x7ffa54101590 msgr2=0x7ffa541039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:23.462 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.786+0000 7ffa5b7be700 1 -- 172.21.15.3:0/3564591125 shutdown_connections 2023-07-20T01:36:23.463 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.786+0000 7ffa5b7be700 1 -- 172.21.15.3:0/3564591125 wait complete. 2023-07-20T01:36:23.463 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.787+0000 7ffa5b7be700 1 Processor -- start 2023-07-20T01:36:23.463 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.787+0000 7ffa5b7be700 1 -- start start 2023-07-20T01:36:23.463 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.787+0000 7ffa5b7be700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa5419bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:23.463 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.787+0000 7ffa5b7be700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5419c480 con 0x7ffa5407acd0 2023-07-20T01:36:23.463 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.787+0000 7ffa5955a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa5419bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:23.464 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.787+0000 7ffa5955a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa5419bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41652/0 (socket says 172.21.15.3:41652) 2023-07-20T01:36:23.464 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.787+0000 7ffa5955a700 1 -- 172.21.15.3:0/383707675 learned_addr learned my addr 172.21.15.3:0/383707675 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:23.464 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.787+0000 7ffa5955a700 1 -- 172.21.15.3:0/383707675 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa50009550 con 0x7ffa5407acd0 2023-07-20T01:36:23.464 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.787+0000 7ffa5955a700 1 --2- 172.21.15.3:0/383707675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa5419bf40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009fb0 tx=0x7ffa50011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:23.464 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.788+0000 7ffa467fc700 1 -- 172.21.15.3:0/383707675 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa50003a40 con 0x7ffa5407acd0 2023-07-20T01:36:23.464 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.788+0000 7ffa467fc700 1 -- 172.21.15.3:0/383707675 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa50003ba0 con 0x7ffa5407acd0 2023-07-20T01:36:23.465 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.788+0000 7ffa5b7be700 1 -- 172.21.15.3:0/383707675 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa5419c680 con 0x7ffa5407acd0 2023-07-20T01:36:23.465 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.788+0000 7ffa467fc700 1 -- 172.21.15.3:0/383707675 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa5001a750 con 0x7ffa5407acd0 2023-07-20T01:36:23.465 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.788+0000 7ffa5b7be700 1 -- 172.21.15.3:0/383707675 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5419caa0 con 0x7ffa5407acd0 2023-07-20T01:36:23.465 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.789+0000 7ffa467fc700 1 -- 172.21.15.3:0/383707675 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ffa50024b60 con 0x7ffa5407acd0 2023-07-20T01:36:23.465 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.789+0000 7ffa5b7be700 1 -- 172.21.15.3:0/383707675 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa54195fc0 con 0x7ffa5407acd0 2023-07-20T01:36:23.465 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.789+0000 7ffa467fc700 1 --2- 172.21.15.3:0/383707675 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7ffa400386c0 0x7ffa4003ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:23.465 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.789+0000 7ffa467fc700 1 -- 172.21.15.3:0/383707675 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ffa5004c7a0 con 0x7ffa5407acd0 2023-07-20T01:36:23.466 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.789+0000 7ffa58d59700 1 --2- 172.21.15.3:0/383707675 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7ffa400386c0 0x7ffa4003ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:23.466 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.790+0000 7ffa58d59700 1 --2- 172.21.15.3:0/383707675 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7ffa400386c0 0x7ffa4003ab80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffa48006fd0 tx=0x7ffa48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:23.466 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.794+0000 7ffa467fc700 1 -- 172.21.15.3:0/383707675 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ffa50024430 con 0x7ffa5407acd0 2023-07-20T01:36:23.466 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:22.889+0000 7ffa467fc700 1 -- 172.21.15.3:0/383707675 <== mon.0 v2:172.21.15.3:3300/0 7 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa5004c1b0 con 0x7ffa5407acd0 2023-07-20T01:36:23.466 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.185+0000 7ffa5b7be700 1 -- 172.21.15.3:0/383707675 --> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ffa54060bb0 con 0x7ffa400386c0 2023-07-20T01:36:23.466 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.192+0000 7ffa467fc700 1 -- 172.21.15.3:0/383707675 <== mgr.14120 v2:172.21.15.3:6800/596448098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ffa54060bb0 con 0x7ffa400386c0 2023-07-20T01:36:23.466 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.195+0000 7ffa5b7be700 1 -- 172.21.15.3:0/383707675 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7ffa400386c0 msgr2=0x7ffa4003ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:23.466 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.195+0000 7ffa5b7be700 1 --2- 172.21.15.3:0/383707675 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7ffa400386c0 0x7ffa4003ab80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffa48006fd0 tx=0x7ffa48006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:23.467 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.195+0000 7ffa5b7be700 1 -- 172.21.15.3:0/383707675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 msgr2=0x7ffa5419bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:23.467 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.195+0000 7ffa5b7be700 1 --2- 172.21.15.3:0/383707675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa5419bf40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009fb0 tx=0x7ffa50011ad0 comp rx=0 tx=0).stop 2023-07-20T01:36:23.467 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.195+0000 7ffa5b7be700 1 -- 172.21.15.3:0/383707675 shutdown_connections 2023-07-20T01:36:23.467 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.195+0000 7ffa5b7be700 1 --2- 172.21.15.3:0/383707675 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7ffa400386c0 0x7ffa4003ab80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:23.467 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.195+0000 7ffa5b7be700 1 --2- 172.21.15.3:0/383707675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa5407acd0 0x7ffa5419bf40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:23.467 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.195+0000 7ffa5b7be700 1 -- 172.21.15.3:0/383707675 >> 172.21.15.3:0/383707675 conn(0x7ffa54101590 msgr2=0x7ffa54102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:23.467 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.195+0000 7ffa5b7be700 1 -- 172.21.15.3:0/383707675 shutdown_connections 2023-07-20T01:36:23.468 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.195+0000 7ffa5b7be700 1 -- 172.21.15.3:0/383707675 wait complete. 2023-07-20T01:36:24.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:23 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:24.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:23 smithi003 ceph-mon[96422]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:36:24.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:23 smithi003 ceph-mon[96422]: Saving service mgr spec with placement count:2 2023-07-20T01:36:24.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:23 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:24.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:23 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:24.811 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.967+0000 7f60a0b9e700 1 Processor -- start 2023-07-20T01:36:24.812 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.967+0000 7f60a0b9e700 1 -- start start 2023-07-20T01:36:24.812 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.968+0000 7f60a0b9e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c105130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:24.812 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.968+0000 7f60a0b9e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c105700 con 0x7f609c104d10 2023-07-20T01:36:24.812 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.968+0000 7f609b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c105130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:24.812 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.968+0000 7f609b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c105130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41656/0 (socket says 172.21.15.3:41656) 2023-07-20T01:36:24.812 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.968+0000 7f609b7fe700 1 -- 172.21.15.3:0/54939241 learned_addr learned my addr 172.21.15.3:0/54939241 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:24.813 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.968+0000 7f609b7fe700 1 -- 172.21.15.3:0/54939241 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c105f10 con 0x7f609c104d10 2023-07-20T01:36:24.813 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.968+0000 7f609b7fe700 1 --2- 172.21.15.3:0/54939241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c105130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f608c0098a0 tx=0x7f608c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fdfacfdec2ec03d server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:24.813 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.968+0000 7f609affd700 1 -- 172.21.15.3:0/54939241 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f608c004160 con 0x7f609c104d10 2023-07-20T01:36:24.813 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.968+0000 7f609affd700 1 -- 172.21.15.3:0/54939241 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f608c0042c0 con 0x7f609c104d10 2023-07-20T01:36:24.813 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.969+0000 7f609affd700 1 -- 172.21.15.3:0/54939241 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f608c0119b0 con 0x7f609c104d10 2023-07-20T01:36:24.813 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.969+0000 7f60a0b9e700 1 -- 172.21.15.3:0/54939241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 msgr2=0x7f609c105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:24.813 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.969+0000 7f60a0b9e700 1 --2- 172.21.15.3:0/54939241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c105130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f608c0098a0 tx=0x7f608c009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:24.814 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.969+0000 7f60a0b9e700 1 -- 172.21.15.3:0/54939241 shutdown_connections 2023-07-20T01:36:24.814 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.969+0000 7f60a0b9e700 1 --2- 172.21.15.3:0/54939241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c105130 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:24.814 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.969+0000 7f60a0b9e700 1 -- 172.21.15.3:0/54939241 >> 172.21.15.3:0/54939241 conn(0x7f609c1002b0 msgr2=0x7f609c1026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:24.814 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.969+0000 7f60a0b9e700 1 -- 172.21.15.3:0/54939241 shutdown_connections 2023-07-20T01:36:24.814 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.969+0000 7f60a0b9e700 1 -- 172.21.15.3:0/54939241 wait complete. 2023-07-20T01:36:24.814 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.970+0000 7f60a0b9e700 1 Processor -- start 2023-07-20T01:36:24.814 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.970+0000 7f60a0b9e700 1 -- start start 2023-07-20T01:36:24.814 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.971+0000 7f60a0b9e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c073c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:24.815 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.971+0000 7f60a0b9e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c19c910 con 0x7f609c104d10 2023-07-20T01:36:24.815 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.971+0000 7f609b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c073c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:24.815 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.971+0000 7f609b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c073c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41658/0 (socket says 172.21.15.3:41658) 2023-07-20T01:36:24.815 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.971+0000 7f609b7fe700 1 -- 172.21.15.3:0/207248665 learned_addr learned my addr 172.21.15.3:0/207248665 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:24.815 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.971+0000 7f609b7fe700 1 -- 172.21.15.3:0/207248665 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608c009550 con 0x7f609c104d10 2023-07-20T01:36:24.815 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.971+0000 7f609b7fe700 1 --2- 172.21.15.3:0/207248665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c073c30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f608c006af0 tx=0x7f608c011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:24.815 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.971+0000 7f60997fa700 1 -- 172.21.15.3:0/207248665 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f608c003a40 con 0x7f609c104d10 2023-07-20T01:36:24.815 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.971+0000 7f60997fa700 1 -- 172.21.15.3:0/207248665 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f608c003ba0 con 0x7f609c104d10 2023-07-20T01:36:24.816 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.972+0000 7f60a0b9e700 1 -- 172.21.15.3:0/207248665 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609c19ca50 con 0x7f609c104d10 2023-07-20T01:36:24.816 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.972+0000 7f60997fa700 1 -- 172.21.15.3:0/207248665 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f608c019690 con 0x7f609c104d10 2023-07-20T01:36:24.816 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.972+0000 7f60a0b9e700 1 -- 172.21.15.3:0/207248665 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609c19ce70 con 0x7f609c104d10 2023-07-20T01:36:24.816 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.973+0000 7f60a0b9e700 1 -- 172.21.15.3:0/207248665 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609c192bb0 con 0x7f609c104d10 2023-07-20T01:36:24.816 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.973+0000 7f60997fa700 1 -- 172.21.15.3:0/207248665 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f608c01aa60 con 0x7f609c104d10 2023-07-20T01:36:24.816 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.974+0000 7f60997fa700 1 --2- 172.21.15.3:0/207248665 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6088038710 0x7f608803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:24.816 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.974+0000 7f60997fa700 1 -- 172.21.15.3:0/207248665 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f608c04cc10 con 0x7f609c104d10 2023-07-20T01:36:24.817 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.974+0000 7f6093fff700 1 --2- 172.21.15.3:0/207248665 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6088038710 0x7f608803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:24.817 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.974+0000 7f6093fff700 1 --2- 172.21.15.3:0/207248665 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6088038710 0x7f608803abd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6084006fd0 tx=0x7f6084006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:24.817 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:23.978+0000 7f60997fa700 1 -- 172.21.15.3:0/207248665 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f608c0237a0 con 0x7f609c104d10 2023-07-20T01:36:24.817 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.343+0000 7f60a0b9e700 1 -- 172.21.15.3:0/207248665 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f609c02c970 con 0x7f609c104d10 2023-07-20T01:36:24.817 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.349+0000 7f60997fa700 1 -- 172.21.15.3:0/207248665 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f608c00b3e0 con 0x7f609c104d10 2023-07-20T01:36:24.817 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.351+0000 7f60a0b9e700 1 -- 172.21.15.3:0/207248665 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6088038710 msgr2=0x7f608803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:24.817 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.351+0000 7f60a0b9e700 1 --2- 172.21.15.3:0/207248665 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6088038710 0x7f608803abd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6084006fd0 tx=0x7f6084006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:24.817 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.351+0000 7f60a0b9e700 1 -- 172.21.15.3:0/207248665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 msgr2=0x7f609c073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:24.818 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.351+0000 7f60a0b9e700 1 --2- 172.21.15.3:0/207248665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c073c30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f608c006af0 tx=0x7f608c011a10 comp rx=0 tx=0).stop 2023-07-20T01:36:24.818 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.352+0000 7f60a0b9e700 1 -- 172.21.15.3:0/207248665 shutdown_connections 2023-07-20T01:36:24.818 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.352+0000 7f60a0b9e700 1 --2- 172.21.15.3:0/207248665 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f6088038710 0x7f608803abd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:24.818 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.352+0000 7f60a0b9e700 1 --2- 172.21.15.3:0/207248665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f609c104d10 0x7f609c073c30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:24.818 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.352+0000 7f60a0b9e700 1 -- 172.21.15.3:0/207248665 >> 172.21.15.3:0/207248665 conn(0x7f609c1002b0 msgr2=0x7f609c101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:24.818 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.352+0000 7f60a0b9e700 1 -- 172.21.15.3:0/207248665 shutdown_connections 2023-07-20T01:36:24.818 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:24.352+0000 7f60a0b9e700 1 -- 172.21.15.3:0/207248665 wait complete. 2023-07-20T01:36:25.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:25 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/207248665' entity='client.admin' 2023-07-20T01:36:25.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:25 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:26.011 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.357+0000 7fb857b72700 1 Processor -- start 2023-07-20T01:36:26.011 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.357+0000 7fb857b72700 1 -- start start 2023-07-20T01:36:26.011 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.357+0000 7fb857b72700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850108e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:26.011 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.357+0000 7fb857b72700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb850073f70 con 0x7fb850106a80 2023-07-20T01:36:26.011 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.358+0000 7fb85590e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850108e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:26.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.358+0000 7fb85590e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850108e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41662/0 (socket says 172.21.15.3:41662) 2023-07-20T01:36:26.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.358+0000 7fb85590e700 1 -- 172.21.15.3:0/1395885750 learned_addr learned my addr 172.21.15.3:0/1395885750 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:26.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.358+0000 7fb85590e700 1 -- 172.21.15.3:0/1395885750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8501093b0 con 0x7fb850106a80 2023-07-20T01:36:26.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.358+0000 7fb85590e700 1 --2- 172.21.15.3:0/1395885750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850108e70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb84c0098a0 tx=0x7fb84c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d0a293fd831e0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:26.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.358+0000 7fb85490c700 1 -- 172.21.15.3:0/1395885750 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb84c004160 con 0x7fb850106a80 2023-07-20T01:36:26.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.358+0000 7fb85490c700 1 -- 172.21.15.3:0/1395885750 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb84c0042c0 con 0x7fb850106a80 2023-07-20T01:36:26.013 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.358+0000 7fb85490c700 1 -- 172.21.15.3:0/1395885750 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb84c0119b0 con 0x7fb850106a80 2023-07-20T01:36:26.013 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.359+0000 7fb857b72700 1 -- 172.21.15.3:0/1395885750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 msgr2=0x7fb850108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:26.013 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.359+0000 7fb857b72700 1 --2- 172.21.15.3:0/1395885750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850108e70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb84c0098a0 tx=0x7fb84c009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:26.013 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.359+0000 7fb857b72700 1 -- 172.21.15.3:0/1395885750 shutdown_connections 2023-07-20T01:36:26.013 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.359+0000 7fb857b72700 1 --2- 172.21.15.3:0/1395885750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850108e70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:26.013 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.359+0000 7fb857b72700 1 -- 172.21.15.3:0/1395885750 >> 172.21.15.3:0/1395885750 conn(0x7fb850100490 msgr2=0x7fb8501028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:26.014 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.359+0000 7fb857b72700 1 -- 172.21.15.3:0/1395885750 shutdown_connections 2023-07-20T01:36:26.014 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.359+0000 7fb857b72700 1 -- 172.21.15.3:0/1395885750 wait complete. 2023-07-20T01:36:26.014 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.360+0000 7fb857b72700 1 Processor -- start 2023-07-20T01:36:26.014 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.360+0000 7fb857b72700 1 -- start start 2023-07-20T01:36:26.015 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.360+0000 7fb857b72700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:26.015 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.360+0000 7fb857b72700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85019a270 con 0x7fb850106a80 2023-07-20T01:36:26.015 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.360+0000 7fb85590e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:26.015 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.360+0000 7fb85590e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41664/0 (socket says 172.21.15.3:41664) 2023-07-20T01:36:26.015 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.360+0000 7fb85590e700 1 -- 172.21.15.3:0/162688441 learned_addr learned my addr 172.21.15.3:0/162688441 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:26.015 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.360+0000 7fb85590e700 1 -- 172.21.15.3:0/162688441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c009550 con 0x7fb850106a80 2023-07-20T01:36:26.016 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.360+0000 7fb85590e700 1 --2- 172.21.15.3:0/162688441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850199d30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb84c003e60 tx=0x7fb84c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:26.016 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.361+0000 7fb846ffd700 1 -- 172.21.15.3:0/162688441 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb84c003970 con 0x7fb850106a80 2023-07-20T01:36:26.016 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.361+0000 7fb846ffd700 1 -- 172.21.15.3:0/162688441 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb84c003ad0 con 0x7fb850106a80 2023-07-20T01:36:26.016 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.361+0000 7fb857b72700 1 -- 172.21.15.3:0/162688441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85019a470 con 0x7fb850106a80 2023-07-20T01:36:26.016 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.361+0000 7fb846ffd700 1 -- 172.21.15.3:0/162688441 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb84c0195e0 con 0x7fb850106a80 2023-07-20T01:36:26.016 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.361+0000 7fb857b72700 1 -- 172.21.15.3:0/162688441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85019a890 con 0x7fb850106a80 2023-07-20T01:36:26.016 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.362+0000 7fb846ffd700 1 -- 172.21.15.3:0/162688441 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fb84c019850 con 0x7fb850106a80 2023-07-20T01:36:26.017 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.362+0000 7fb846ffd700 1 --2- 172.21.15.3:0/162688441 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fb83c038710 0x7fb83c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:26.017 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.362+0000 7fb857b72700 1 -- 172.21.15.3:0/162688441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb850061da0 con 0x7fb850106a80 2023-07-20T01:36:26.017 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.362+0000 7fb846ffd700 1 -- 172.21.15.3:0/162688441 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb84c04ca20 con 0x7fb850106a80 2023-07-20T01:36:26.017 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.362+0000 7fb85510d700 1 --2- 172.21.15.3:0/162688441 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fb83c038710 0x7fb83c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:26.017 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.363+0000 7fb85510d700 1 --2- 172.21.15.3:0/162688441 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fb83c038710 0x7fb83c03abd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb840006fd0 tx=0x7fb840006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:26.017 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.367+0000 7fb846ffd700 1 -- 172.21.15.3:0/162688441 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb84c019b50 con 0x7fb850106a80 2023-07-20T01:36:26.017 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.737+0000 7fb857b72700 1 -- 172.21.15.3:0/162688441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb85019d2d0 con 0x7fb850106a80 2023-07-20T01:36:26.018 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.743+0000 7fb846ffd700 1 -- 172.21.15.3:0/162688441 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb84c018b40 con 0x7fb850106a80 2023-07-20T01:36:26.018 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.746+0000 7fb857b72700 1 -- 172.21.15.3:0/162688441 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fb83c038710 msgr2=0x7fb83c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:26.018 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.746+0000 7fb857b72700 1 --2- 172.21.15.3:0/162688441 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fb83c038710 0x7fb83c03abd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb840006fd0 tx=0x7fb840006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:26.018 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.746+0000 7fb857b72700 1 -- 172.21.15.3:0/162688441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 msgr2=0x7fb850199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:26.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.746+0000 7fb857b72700 1 --2- 172.21.15.3:0/162688441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850199d30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb84c003e60 tx=0x7fb84c011aa0 comp rx=0 tx=0).stop 2023-07-20T01:36:26.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.746+0000 7fb857b72700 1 -- 172.21.15.3:0/162688441 shutdown_connections 2023-07-20T01:36:26.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.746+0000 7fb857b72700 1 --2- 172.21.15.3:0/162688441 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7fb83c038710 0x7fb83c03abd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:26.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.746+0000 7fb857b72700 1 --2- 172.21.15.3:0/162688441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb850106a80 0x7fb850199d30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:26.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.746+0000 7fb857b72700 1 -- 172.21.15.3:0/162688441 >> 172.21.15.3:0/162688441 conn(0x7fb850100490 msgr2=0x7fb850101b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:26.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.746+0000 7fb857b72700 1 -- 172.21.15.3:0/162688441 shutdown_connections 2023-07-20T01:36:26.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:25.746+0000 7fb857b72700 1 -- 172.21.15.3:0/162688441 wait complete. 2023-07-20T01:36:26.020 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling the dashboard module... 2023-07-20T01:36:27.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:26 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/162688441' entity='client.admin' 2023-07-20T01:36:27.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:26 smithi003 ceph-mon[96422]: from='mgr.14120 172.21.15.3:0/3660939122' entity='mgr.x' 2023-07-20T01:36:28.025 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:27 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3922085681' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-20T01:36:28.026 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:27 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: ignoring --setuser ceph since I am not root 2023-07-20T01:36:28.026 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:27 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: ignoring --setgroup ceph since I am not root 2023-07-20T01:36:28.026 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:27 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:27.957+0000 7fbe089b5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T01:36:28.026 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:28 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:28.053+0000 7fbe089b5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T01:36:28.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.569+0000 7f3dad512700 1 Processor -- start 2023-07-20T01:36:28.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.569+0000 7f3dad512700 1 -- start start 2023-07-20T01:36:28.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.569+0000 7f3dad512700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8104670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:28.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.569+0000 7f3dad512700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da8104c40 con 0x7f3da8104250 2023-07-20T01:36:28.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.569+0000 7f3da6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:28.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.569+0000 7f3da6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8104670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41668/0 (socket says 172.21.15.3:41668) 2023-07-20T01:36:28.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.569+0000 7f3da6ffd700 1 -- 172.21.15.3:0/39520994 learned_addr learned my addr 172.21.15.3:0/39520994 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:28.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.570+0000 7f3da6ffd700 1 -- 172.21.15.3:0/39520994 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da81071e0 con 0x7f3da8104250 2023-07-20T01:36:28.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.570+0000 7f3da6ffd700 1 --2- 172.21.15.3:0/39520994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8104670 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3d980098a0 tx=0x7f3d98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a941631f6349f5 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:28.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.570+0000 7f3da5ffb700 1 -- 172.21.15.3:0/39520994 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d98004160 con 0x7f3da8104250 2023-07-20T01:36:28.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.570+0000 7f3da5ffb700 1 -- 172.21.15.3:0/39520994 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d980042c0 con 0x7f3da8104250 2023-07-20T01:36:28.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.570+0000 7f3da5ffb700 1 -- 172.21.15.3:0/39520994 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d980119b0 con 0x7f3da8104250 2023-07-20T01:36:28.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.570+0000 7f3dad512700 1 -- 172.21.15.3:0/39520994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 msgr2=0x7f3da8104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:28.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.570+0000 7f3dad512700 1 --2- 172.21.15.3:0/39520994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8104670 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3d980098a0 tx=0x7f3d98009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:28.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.571+0000 7f3dad512700 1 -- 172.21.15.3:0/39520994 shutdown_connections 2023-07-20T01:36:28.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.571+0000 7f3dad512700 1 --2- 172.21.15.3:0/39520994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8104670 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:28.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.571+0000 7f3dad512700 1 -- 172.21.15.3:0/39520994 >> 172.21.15.3:0/39520994 conn(0x7f3da80ffbc0 msgr2=0x7f3da8101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:28.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.571+0000 7f3dad512700 1 -- 172.21.15.3:0/39520994 shutdown_connections 2023-07-20T01:36:28.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.571+0000 7f3dad512700 1 -- 172.21.15.3:0/39520994 wait complete. 2023-07-20T01:36:28.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.571+0000 7f3dad512700 1 Processor -- start 2023-07-20T01:36:28.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.571+0000 7f3dad512700 1 -- start start 2023-07-20T01:36:28.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3dad512700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:28.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3dad512700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da819a170 con 0x7f3da8104250 2023-07-20T01:36:28.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3da6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:28.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3da6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41670/0 (socket says 172.21.15.3:41670) 2023-07-20T01:36:28.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3da6ffd700 1 -- 172.21.15.3:0/3922085681 learned_addr learned my addr 172.21.15.3:0/3922085681 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:28.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3da6ffd700 1 -- 172.21.15.3:0/3922085681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d98009550 con 0x7f3da8104250 2023-07-20T01:36:28.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3da6ffd700 1 --2- 172.21.15.3:0/3922085681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8199c30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3d98000c00 tx=0x7f3d98011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:28.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3d8ffff700 1 -- 172.21.15.3:0/3922085681 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d980039f0 con 0x7f3da8104250 2023-07-20T01:36:28.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3d8ffff700 1 -- 172.21.15.3:0/3922085681 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d98003b50 con 0x7f3da8104250 2023-07-20T01:36:28.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3dad512700 1 -- 172.21.15.3:0/3922085681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da819a370 con 0x7f3da8104250 2023-07-20T01:36:28.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3d8ffff700 1 -- 172.21.15.3:0/3922085681 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d9801a780 con 0x7f3da8104250 2023-07-20T01:36:28.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.572+0000 7f3dad512700 1 -- 172.21.15.3:0/3922085681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da819a790 con 0x7f3da8104250 2023-07-20T01:36:28.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.573+0000 7f3d8ffff700 1 -- 172.21.15.3:0/3922085681 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f3d9801aa40 con 0x7f3da8104250 2023-07-20T01:36:28.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.573+0000 7f3d8ffff700 1 --2- 172.21.15.3:0/3922085681 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f3d90038710 0x7f3d9003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:28.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.573+0000 7f3d8ffff700 1 -- 172.21.15.3:0/3922085681 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3d9804c9a0 con 0x7f3da8104250 2023-07-20T01:36:28.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.573+0000 7f3dad512700 1 -- 172.21.15.3:0/3922085681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da810a3d0 con 0x7f3da8104250 2023-07-20T01:36:28.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.574+0000 7f3da67fc700 1 --2- 172.21.15.3:0/3922085681 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f3d90038710 0x7f3d9003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:28.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.574+0000 7f3da67fc700 1 --2- 172.21.15.3:0/3922085681 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f3d90038710 0x7f3d9003abd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c006fd0 tx=0x7f3d9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:28.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:26.578+0000 7f3d8ffff700 1 -- 172.21.15.3:0/3922085681 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3d9801acf0 con 0x7f3da8104250 2023-07-20T01:36:28.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.059+0000 7f3dad512700 1 -- 172.21.15.3:0/3922085681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f3da8105830 con 0x7f3da8104250 2023-07-20T01:36:28.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.754+0000 7f3d8ffff700 1 -- 172.21.15.3:0/3922085681 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f3d9802dc90 con 0x7f3da8104250 2023-07-20T01:36:28.159 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.755+0000 7f3d8ffff700 1 -- 172.21.15.3:0/3922085681 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f3d9804c2f0 con 0x7f3da8104250 2023-07-20T01:36:28.159 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.758+0000 7f3dad512700 1 -- 172.21.15.3:0/3922085681 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f3d90038710 msgr2=0x7f3d9003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:28.159 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.758+0000 7f3dad512700 1 --2- 172.21.15.3:0/3922085681 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f3d90038710 0x7f3d9003abd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c006fd0 tx=0x7f3d9c006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:28.159 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.758+0000 7f3dad512700 1 -- 172.21.15.3:0/3922085681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 msgr2=0x7f3da8199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:28.159 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.758+0000 7f3dad512700 1 --2- 172.21.15.3:0/3922085681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8199c30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3d98000c00 tx=0x7f3d98011a30 comp rx=0 tx=0).stop 2023-07-20T01:36:28.160 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.758+0000 7f3dad512700 1 -- 172.21.15.3:0/3922085681 shutdown_connections 2023-07-20T01:36:28.160 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.758+0000 7f3dad512700 1 --2- 172.21.15.3:0/3922085681 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f3d90038710 0x7f3d9003abd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:28.160 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.758+0000 7f3dad512700 1 --2- 172.21.15.3:0/3922085681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3da8104250 0x7f3da8199c30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:28.160 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.758+0000 7f3dad512700 1 -- 172.21.15.3:0/3922085681 >> 172.21.15.3:0/3922085681 conn(0x7f3da80ffbc0 msgr2=0x7f3da8100920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:28.160 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.758+0000 7f3dad512700 1 -- 172.21.15.3:0/3922085681 shutdown_connections 2023-07-20T01:36:28.160 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:27.758+0000 7f3dad512700 1 -- 172.21.15.3:0/3922085681 wait complete. 2023-07-20T01:36:28.352 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:28 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:28.162+0000 7fbe089b5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T01:36:28.352 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:28 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:28.282+0000 7fbe089b5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T01:36:28.690 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:28 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:28.496+0000 7fbe089b5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T01:36:28.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:28 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3922085681' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-20T01:36:28.980 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:28 smithi003 ceph-mon[96422]: mgrmap e9: x(active, since 17s) 2023-07-20T01:36:28.980 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:28 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:28.899+0000 7fbe089b5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T01:36:29.252 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:29 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:29.008+0000 7fbe089b5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T01:36:29.252 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:29 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:29.159+0000 7fbe089b5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T01:36:29.501 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:36:29.502 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-20T01:36:29.502 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T01:36:29.502 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T01:36:29.503 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T01:36:29.503 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:36:29.503 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.695+0000 7f26336fa700 1 Processor -- start 2023-07-20T01:36:29.503 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.696+0000 7f26336fa700 1 -- start start 2023-07-20T01:36:29.504 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.696+0000 7f26336fa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c106410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:29.504 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.696+0000 7f26336fa700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c1069e0 con 0x7f262c105ff0 2023-07-20T01:36:29.504 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.696+0000 7f2631496700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c106410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:29.505 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.696+0000 7f2631496700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c106410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41684/0 (socket says 172.21.15.3:41684) 2023-07-20T01:36:29.505 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.696+0000 7f2631496700 1 -- 172.21.15.3:0/2335135307 learned_addr learned my addr 172.21.15.3:0/2335135307 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:29.505 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.696+0000 7f2631496700 1 -- 172.21.15.3:0/2335135307 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262c1071f0 con 0x7f262c105ff0 2023-07-20T01:36:29.506 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.697+0000 7f2631496700 1 --2- 172.21.15.3:0/2335135307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c106410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f26280098a0 tx=0x7f2628009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6045d4f39029d3e2 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:29.506 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.697+0000 7f2623fff700 1 -- 172.21.15.3:0/2335135307 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2628004160 con 0x7f262c105ff0 2023-07-20T01:36:29.506 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.697+0000 7f2623fff700 1 -- 172.21.15.3:0/2335135307 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26280042c0 con 0x7f262c105ff0 2023-07-20T01:36:29.506 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.697+0000 7f2623fff700 1 -- 172.21.15.3:0/2335135307 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26280119b0 con 0x7f262c105ff0 2023-07-20T01:36:29.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.698+0000 7f26336fa700 1 -- 172.21.15.3:0/2335135307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 msgr2=0x7f262c106410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:29.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.698+0000 7f26336fa700 1 --2- 172.21.15.3:0/2335135307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c106410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f26280098a0 tx=0x7f2628009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:29.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.698+0000 7f26336fa700 1 -- 172.21.15.3:0/2335135307 shutdown_connections 2023-07-20T01:36:29.508 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.698+0000 7f26336fa700 1 --2- 172.21.15.3:0/2335135307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c106410 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:29.508 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.698+0000 7f26336fa700 1 -- 172.21.15.3:0/2335135307 >> 172.21.15.3:0/2335135307 conn(0x7f262c101590 msgr2=0x7f262c1039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:29.508 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.698+0000 7f26336fa700 1 -- 172.21.15.3:0/2335135307 shutdown_connections 2023-07-20T01:36:29.509 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.698+0000 7f26336fa700 1 -- 172.21.15.3:0/2335135307 wait complete. 2023-07-20T01:36:29.509 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.699+0000 7f26336fa700 1 Processor -- start 2023-07-20T01:36:29.509 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.699+0000 7f26336fa700 1 -- start start 2023-07-20T01:36:29.509 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.699+0000 7f26336fa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:29.510 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.699+0000 7f26336fa700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c1980f0 con 0x7f262c105ff0 2023-07-20T01:36:29.510 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.699+0000 7f2631496700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:29.510 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.699+0000 7f2631496700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41686/0 (socket says 172.21.15.3:41686) 2023-07-20T01:36:29.510 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.699+0000 7f2631496700 1 -- 172.21.15.3:0/3180639969 learned_addr learned my addr 172.21.15.3:0/3180639969 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:29.510 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.699+0000 7f2631496700 1 -- 172.21.15.3:0/3180639969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2628009550 con 0x7f262c105ff0 2023-07-20T01:36:29.511 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.699+0000 7f2631496700 1 --2- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c197bb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2628006af0 tx=0x7f2628011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:29.511 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.700+0000 7f26227fc700 1 -- 172.21.15.3:0/3180639969 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2628003a40 con 0x7f262c105ff0 2023-07-20T01:36:29.511 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.700+0000 7f26336fa700 1 -- 172.21.15.3:0/3180639969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262c1982f0 con 0x7f262c105ff0 2023-07-20T01:36:29.511 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.700+0000 7f26227fc700 1 -- 172.21.15.3:0/3180639969 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2628003ba0 con 0x7f262c105ff0 2023-07-20T01:36:29.512 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.700+0000 7f26227fc700 1 -- 172.21.15.3:0/3180639969 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2628019690 con 0x7f262c105ff0 2023-07-20T01:36:29.512 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.700+0000 7f26336fa700 1 -- 172.21.15.3:0/3180639969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262c198710 con 0x7f262c105ff0 2023-07-20T01:36:29.512 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.701+0000 7f26227fc700 1 -- 172.21.15.3:0/3180639969 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f2628019950 con 0x7f262c105ff0 2023-07-20T01:36:29.513 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.701+0000 7f26227fc700 1 --2- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f261803cb10 0x7f261803efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:29.513 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.701+0000 7f26336fa700 1 -- 172.21.15.3:0/3180639969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262c1918e0 con 0x7f262c105ff0 2023-07-20T01:36:29.513 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.701+0000 7f2630c95700 1 -- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f261803cb10 msgr2=0x7f261803efd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/596448098 2023-07-20T01:36:29.513 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.701+0000 7f26227fc700 1 -- 172.21.15.3:0/3180639969 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f262804cb60 con 0x7f262c105ff0 2023-07-20T01:36:29.513 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.701+0000 7f2630c95700 1 --2- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f261803cb10 0x7f261803efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:36:29.513 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.710+0000 7f26227fc700 1 -- 172.21.15.3:0/3180639969 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f262802b750 con 0x7f262c105ff0 2023-07-20T01:36:29.513 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.902+0000 7f2630c95700 1 -- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f261803cb10 msgr2=0x7f261803efd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/596448098 2023-07-20T01:36:29.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:28.902+0000 7f2630c95700 1 --2- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f261803cb10 0x7f261803efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T01:36:29.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.179+0000 7f26336fa700 1 -- 172.21.15.3:0/3180639969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f262c02cd90 con 0x7f262c105ff0 2023-07-20T01:36:29.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.179+0000 7f26227fc700 1 -- 172.21.15.3:0/3180639969 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f2628018350 con 0x7f262c105ff0 2023-07-20T01:36:29.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.182+0000 7f26336fa700 1 -- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f261803cb10 msgr2=0x7f261803efd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:36:29.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.182+0000 7f26336fa700 1 --2- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f261803cb10 0x7f261803efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:29.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.182+0000 7f26336fa700 1 -- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 msgr2=0x7f262c197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:29.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.182+0000 7f26336fa700 1 --2- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c197bb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2628006af0 tx=0x7f2628011a10 comp rx=0 tx=0).stop 2023-07-20T01:36:29.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.182+0000 7f26336fa700 1 -- 172.21.15.3:0/3180639969 shutdown_connections 2023-07-20T01:36:29.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.182+0000 7f26336fa700 1 --2- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f261803cb10 0x7f261803efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:29.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.182+0000 7f26336fa700 1 --2- 172.21.15.3:0/3180639969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c105ff0 0x7f262c197bb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:29.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.182+0000 7f26336fa700 1 -- 172.21.15.3:0/3180639969 >> 172.21.15.3:0/3180639969 conn(0x7f262c101590 msgr2=0x7f262c103030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:29.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.182+0000 7f26336fa700 1 -- 172.21.15.3:0/3180639969 shutdown_connections 2023-07-20T01:36:29.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.182+0000 7f26336fa700 1 -- 172.21.15.3:0/3180639969 wait complete. 2023-07-20T01:36:29.515 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2023-07-20T01:36:29.516 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 9... 2023-07-20T01:36:29.659 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:29 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:29.281+0000 7fbe089b5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T01:36:29.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:29 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3180639969' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T01:36:29.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:29 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:29.806+0000 7fbe089b5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T01:36:30.939 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:30 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:30.491+0000 7fbe089b5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T01:36:30.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:30 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:30.683+0000 7fbe089b5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T01:36:31.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:31 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:31.643+0000 7fbe089b5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:36:31.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:31 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:31.754+0000 7fbe089b5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T01:36:32.293 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:31 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:31.973+0000 7fbe089b5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:36:32.547 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:32 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:32.322+0000 7fbe089b5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T01:36:32.939 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:32 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:32.575+0000 7fbe089b5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T01:36:32.940 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:32 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:32.757+0000 7fbe089b5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T01:36:33.439 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:33 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:33.085+0000 7fbe089b5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T01:36:34.189 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:33 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:33.906+0000 7fbe089b5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T01:36:34.190 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:34 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:34.064+0000 7fbe089b5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T01:36:34.689 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:34 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:34.308+0000 7fbe089b5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T01:36:34.690 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:34 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:34.445+0000 7fbe089b5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T01:36:35.189 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:34 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:34.718+0000 7fbe089b5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T01:36:36.189 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:36:35.758+0000 7fbe089b5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T01:36:36.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: Active manager daemon x restarted 2023-07-20T01:36:36.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: Activating manager daemon x 2023-07-20T01:36:36.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: osdmap e3: 0 total, 0 up, 0 in 2023-07-20T01:36:36.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: mgrmap e10: x(active, starting, since 0.00387029s) 2023-07-20T01:36:36.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:36:36.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T01:36:36.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T01:36:36.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T01:36:36.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T01:36:36.192 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: Manager daemon x is now available 2023-07-20T01:36:36.192 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:36:36.192 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:36:37.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T01:36:37.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T01:36:37.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:36 smithi003 ceph-mon[96422]: mgrmap e11: x(active, since 1.00579s) 2023-07-20T01:36:37.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:36 smithi003 ceph-mon[96422]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T01:36:37.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:36 smithi003 ceph-mon[96422]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T01:36:37.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:36:37.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-20T01:36:37.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T01:36:37.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:36:37.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.980+0000 7f4b52346700 1 Processor -- start 2023-07-20T01:36:37.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.981+0000 7f4b52346700 1 -- start start 2023-07-20T01:36:37.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.981+0000 7f4b52346700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c104cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:37.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.981+0000 7f4b52346700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c073f30 con 0x7f4b4c1048b0 2023-07-20T01:36:37.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.981+0000 7f4b4bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c104cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:37.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.981+0000 7f4b4bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c104cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41694/0 (socket says 172.21.15.3:41694) 2023-07-20T01:36:37.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.981+0000 7f4b4bfff700 1 -- 172.21.15.3:0/2762461098 learned_addr learned my addr 172.21.15.3:0/2762461098 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:37.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.981+0000 7f4b4bfff700 1 -- 172.21.15.3:0/2762461098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4c105210 con 0x7f4b4c1048b0 2023-07-20T01:36:37.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.982+0000 7f4b4bfff700 1 --2- 172.21.15.3:0/2762461098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c104cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0098a0 tx=0x7f4b3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54eda39627686912 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:37.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.982+0000 7f4b4affd700 1 -- 172.21.15.3:0/2762461098 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b3c004160 con 0x7f4b4c1048b0 2023-07-20T01:36:37.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.982+0000 7f4b4affd700 1 -- 172.21.15.3:0/2762461098 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b3c0042c0 con 0x7f4b4c1048b0 2023-07-20T01:36:37.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.982+0000 7f4b4affd700 1 -- 172.21.15.3:0/2762461098 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b3c0119b0 con 0x7f4b4c1048b0 2023-07-20T01:36:37.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.982+0000 7f4b52346700 1 -- 172.21.15.3:0/2762461098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 msgr2=0x7f4b4c104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:37.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.982+0000 7f4b52346700 1 --2- 172.21.15.3:0/2762461098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c104cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0098a0 tx=0x7f4b3c009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:37.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.983+0000 7f4b52346700 1 -- 172.21.15.3:0/2762461098 shutdown_connections 2023-07-20T01:36:37.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.983+0000 7f4b52346700 1 --2- 172.21.15.3:0/2762461098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c104cd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:37.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.983+0000 7f4b52346700 1 -- 172.21.15.3:0/2762461098 >> 172.21.15.3:0/2762461098 conn(0x7f4b4c100490 msgr2=0x7f4b4c1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:37.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.983+0000 7f4b52346700 1 -- 172.21.15.3:0/2762461098 shutdown_connections 2023-07-20T01:36:37.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.983+0000 7f4b52346700 1 -- 172.21.15.3:0/2762461098 wait complete. 2023-07-20T01:36:37.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.983+0000 7f4b52346700 1 Processor -- start 2023-07-20T01:36:37.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.983+0000 7f4b52346700 1 -- start start 2023-07-20T01:36:37.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.984+0000 7f4b52346700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c19bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:37.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.984+0000 7f4b52346700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c19c450 con 0x7f4b4c1048b0 2023-07-20T01:36:37.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.984+0000 7f4b4bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c19bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:37.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.984+0000 7f4b4bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c19bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41696/0 (socket says 172.21.15.3:41696) 2023-07-20T01:36:37.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.984+0000 7f4b4bfff700 1 -- 172.21.15.3:0/917105984 learned_addr learned my addr 172.21.15.3:0/917105984 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:37.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.984+0000 7f4b4bfff700 1 -- 172.21.15.3:0/917105984 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c009550 con 0x7f4b4c1048b0 2023-07-20T01:36:37.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.984+0000 7f4b4bfff700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c19bf10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c003e60 tx=0x7f4b3c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:37.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.984+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b3c003970 con 0x7f4b4c1048b0 2023-07-20T01:36:37.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.984+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b3c003ad0 con 0x7f4b4c1048b0 2023-07-20T01:36:37.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.984+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b3c0195e0 con 0x7f4b4c1048b0 2023-07-20T01:36:37.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.984+0000 7f4b52346700 1 -- 172.21.15.3:0/917105984 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4c19c650 con 0x7f4b4c1048b0 2023-07-20T01:36:37.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.985+0000 7f4b52346700 1 -- 172.21.15.3:0/917105984 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4c19ca70 con 0x7f4b4c1048b0 2023-07-20T01:36:37.268 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.986+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f4b3c0198a0 con 0x7f4b4c1048b0 2023-07-20T01:36:37.268 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.986+0000 7f4b497fa700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 0x7f4b3403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:37.268 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.986+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 --> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b3403b2e0 con 0x7f4b34038710 2023-07-20T01:36:37.268 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.986+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4b3c04c8d0 con 0x7f4b4c1048b0 2023-07-20T01:36:37.268 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.986+0000 7f4b4b7fe700 1 -- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 msgr2=0x7f4b3403abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/596448098 2023-07-20T01:36:37.268 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:29.986+0000 7f4b4b7fe700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 0x7f4b3403abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:36:37.269 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:30.187+0000 7f4b4b7fe700 1 -- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 msgr2=0x7f4b3403abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/596448098 2023-07-20T01:36:37.269 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:30.187+0000 7f4b4b7fe700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 0x7f4b3403abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T01:36:37.269 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:30.588+0000 7f4b4b7fe700 1 -- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 msgr2=0x7f4b3403abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/596448098 2023-07-20T01:36:37.269 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:30.588+0000 7f4b4b7fe700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 0x7f4b3403abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T01:36:37.269 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:31.389+0000 7f4b4b7fe700 1 -- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 msgr2=0x7f4b3403abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/596448098 2023-07-20T01:36:37.269 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:31.389+0000 7f4b4b7fe700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 0x7f4b3403abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T01:36:37.269 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:32.990+0000 7f4b4b7fe700 1 -- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 msgr2=0x7f4b3403abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/596448098 2023-07-20T01:36:37.269 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:32.990+0000 7f4b4b7fe700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 0x7f4b3403abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T01:36:37.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:35.773+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f4b3c02da00 con 0x7f4b4c1048b0 2023-07-20T01:36:37.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:35.773+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 msgr2=0x7f4b3403abd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:36:37.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:35.773+0000 7f4b497fa700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 0x7f4b3403abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:37.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.775+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f4b3c04e880 con 0x7f4b4c1048b0 2023-07-20T01:36:37.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.775+0000 7f4b497fa700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4b3403bfb0 0x7f4b3403e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:37.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.775+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b3403b2e0 con 0x7f4b3403bfb0 2023-07-20T01:36:37.270 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.775+0000 7f4b4b7fe700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4b3403bfb0 0x7f4b3403e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:37.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.776+0000 7f4b4b7fe700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4b3403bfb0 0x7f4b3403e3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4b40003a10 tx=0x7f4b400092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:37.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.777+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f4b3403b2e0 con 0x7f4b3403bfb0 2023-07-20T01:36:37.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b52346700 1 -- 172.21.15.3:0/917105984 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4b4c19d250 con 0x7f4b3403bfb0 2023-07-20T01:36:37.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b497fa700 1 -- 172.21.15.3:0/917105984 <== mgr.14152 v2:172.21.15.3:6800/4201887131 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f4b4c19d250 con 0x7f4b3403bfb0 2023-07-20T01:36:37.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b52346700 1 -- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4b3403bfb0 msgr2=0x7f4b3403e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:37.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b52346700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4b3403bfb0 0x7f4b3403e3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4b40003a10 tx=0x7f4b400092b0 comp rx=0 tx=0).stop 2023-07-20T01:36:37.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b52346700 1 -- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 msgr2=0x7f4b4c19bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:37.271 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b52346700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c19bf10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c003e60 tx=0x7f4b3c011aa0 comp rx=0 tx=0).stop 2023-07-20T01:36:37.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b52346700 1 -- 172.21.15.3:0/917105984 shutdown_connections 2023-07-20T01:36:37.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b52346700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4b3403bfb0 0x7f4b3403e3a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:37.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b52346700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b4c1048b0 0x7f4b4c19bf10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:37.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b52346700 1 --2- 172.21.15.3:0/917105984 >> [v2:172.21.15.3:6800/596448098,v1:172.21.15.3:6801/596448098] conn(0x7f4b34038710 0x7f4b3403abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:37.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b52346700 1 -- 172.21.15.3:0/917105984 >> 172.21.15.3:0/917105984 conn(0x7f4b4c100490 msgr2=0x7f4b4c190470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:37.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.788+0000 7f4b52346700 1 -- 172.21.15.3:0/917105984 shutdown_connections 2023-07-20T01:36:37.272 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:36.789+0000 7f4b52346700 1 -- 172.21.15.3:0/917105984 wait complete. 2023-07-20T01:36:37.273 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 9 is available 2023-07-20T01:36:37.273 INFO:teuthology.orchestra.run.smithi003.stdout:Generating a dashboard self-signed certificate... 2023-07-20T01:36:38.509 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-20T01:36:38.509 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.779+0000 7f3e0614d700 1 Processor -- start 2023-07-20T01:36:38.510 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.779+0000 7f3e0614d700 1 -- start start 2023-07-20T01:36:38.510 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.779+0000 7f3e0614d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00108e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:38.510 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.779+0000 7f3e0614d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e00073fb0 con 0x7f3e00106a40 2023-07-20T01:36:38.510 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.780+0000 7f3dff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00108e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:38.510 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.780+0000 7f3dff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00108e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41750/0 (socket says 172.21.15.3:41750) 2023-07-20T01:36:38.511 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.780+0000 7f3dff7fe700 1 -- 172.21.15.3:0/3816251866 learned_addr learned my addr 172.21.15.3:0/3816251866 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:38.511 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.780+0000 7f3dff7fe700 1 -- 172.21.15.3:0/3816251866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00109370 con 0x7f3e00106a40 2023-07-20T01:36:38.511 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.780+0000 7f3dff7fe700 1 --2- 172.21.15.3:0/3816251866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00108e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3df00098a0 tx=0x7f3df0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8a9ffaa5f17686 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:38.511 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.780+0000 7f3dfe7fc700 1 -- 172.21.15.3:0/3816251866 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3df0004160 con 0x7f3e00106a40 2023-07-20T01:36:38.511 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.780+0000 7f3dfe7fc700 1 -- 172.21.15.3:0/3816251866 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3df00042c0 con 0x7f3e00106a40 2023-07-20T01:36:38.512 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.780+0000 7f3dfe7fc700 1 -- 172.21.15.3:0/3816251866 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3df00119b0 con 0x7f3e00106a40 2023-07-20T01:36:38.512 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.781+0000 7f3e0614d700 1 -- 172.21.15.3:0/3816251866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 msgr2=0x7f3e00108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:38.512 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.781+0000 7f3e0614d700 1 --2- 172.21.15.3:0/3816251866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00108e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3df00098a0 tx=0x7f3df0009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:38.512 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.781+0000 7f3e0614d700 1 -- 172.21.15.3:0/3816251866 shutdown_connections 2023-07-20T01:36:38.513 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.781+0000 7f3e0614d700 1 --2- 172.21.15.3:0/3816251866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00108e30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:38.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.781+0000 7f3e0614d700 1 -- 172.21.15.3:0/3816251866 >> 172.21.15.3:0/3816251866 conn(0x7f3e00100490 msgr2=0x7f3e001028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:38.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.781+0000 7f3e0614d700 1 -- 172.21.15.3:0/3816251866 shutdown_connections 2023-07-20T01:36:38.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.781+0000 7f3e0614d700 1 -- 172.21.15.3:0/3816251866 wait complete. 2023-07-20T01:36:38.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.782+0000 7f3e0614d700 1 Processor -- start 2023-07-20T01:36:38.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.782+0000 7f3e0614d700 1 -- start start 2023-07-20T01:36:38.514 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.782+0000 7f3e0614d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:38.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.782+0000 7f3e0614d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0019a250 con 0x7f3e00106a40 2023-07-20T01:36:38.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.782+0000 7f3dff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:38.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.783+0000 7f3dff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41752/0 (socket says 172.21.15.3:41752) 2023-07-20T01:36:38.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.783+0000 7f3dff7fe700 1 -- 172.21.15.3:0/3555627116 learned_addr learned my addr 172.21.15.3:0/3555627116 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:38.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.783+0000 7f3dff7fe700 1 -- 172.21.15.3:0/3555627116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df0009550 con 0x7f3e00106a40 2023-07-20T01:36:38.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.783+0000 7f3dff7fe700 1 --2- 172.21.15.3:0/3555627116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00199d10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3df0003e60 tx=0x7f3df0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:38.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.783+0000 7f3dfcff9700 1 -- 172.21.15.3:0/3555627116 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3df0003970 con 0x7f3e00106a40 2023-07-20T01:36:38.515 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.783+0000 7f3dfcff9700 1 -- 172.21.15.3:0/3555627116 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3df0003ad0 con 0x7f3e00106a40 2023-07-20T01:36:38.516 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.783+0000 7f3e0614d700 1 -- 172.21.15.3:0/3555627116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0019a450 con 0x7f3e00106a40 2023-07-20T01:36:38.516 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.783+0000 7f3dfcff9700 1 -- 172.21.15.3:0/3555627116 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3df00195e0 con 0x7f3e00106a40 2023-07-20T01:36:38.516 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.783+0000 7f3e0614d700 1 -- 172.21.15.3:0/3555627116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0019a870 con 0x7f3e00106a40 2023-07-20T01:36:38.516 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.784+0000 7f3dfcff9700 1 -- 172.21.15.3:0/3555627116 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f3df00198a0 con 0x7f3e00106a40 2023-07-20T01:36:38.516 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.785+0000 7f3dfcff9700 1 --2- 172.21.15.3:0/3555627116 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3de80385f0 0x7f3de803aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:38.516 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.785+0000 7f3dfcff9700 1 -- 172.21.15.3:0/3555627116 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3df004ca80 con 0x7f3e00106a40 2023-07-20T01:36:38.516 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.785+0000 7f3e0614d700 1 -- 172.21.15.3:0/3555627116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e00061da0 con 0x7f3e00106a40 2023-07-20T01:36:38.517 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.785+0000 7f3dfeffd700 1 --2- 172.21.15.3:0/3555627116 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3de80385f0 0x7f3de803aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:38.517 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.785+0000 7f3dfeffd700 1 --2- 172.21.15.3:0/3555627116 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3de80385f0 0x7f3de803aab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3df4006fd0 tx=0x7f3df4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:38.517 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:37.790+0000 7f3dfcff9700 1 -- 172.21.15.3:0/3555627116 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3df0019bf0 con 0x7f3e00106a40 2023-07-20T01:36:38.517 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.164+0000 7f3e0614d700 1 -- 172.21.15.3:0/3555627116 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e00102790 con 0x7f3de80385f0 2023-07-20T01:36:38.517 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.193+0000 7f3dfcff9700 1 -- 172.21.15.3:0/3555627116 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3e00102790 con 0x7f3de80385f0 2023-07-20T01:36:38.517 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.196+0000 7f3e0614d700 1 -- 172.21.15.3:0/3555627116 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3de80385f0 msgr2=0x7f3de803aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:38.517 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.196+0000 7f3e0614d700 1 --2- 172.21.15.3:0/3555627116 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3de80385f0 0x7f3de803aab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3df4006fd0 tx=0x7f3df4006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:38.518 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.196+0000 7f3e0614d700 1 -- 172.21.15.3:0/3555627116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 msgr2=0x7f3e00199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:38.518 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.196+0000 7f3e0614d700 1 --2- 172.21.15.3:0/3555627116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00199d10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3df0003e60 tx=0x7f3df0011aa0 comp rx=0 tx=0).stop 2023-07-20T01:36:38.518 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.196+0000 7f3e0614d700 1 -- 172.21.15.3:0/3555627116 shutdown_connections 2023-07-20T01:36:38.518 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.196+0000 7f3e0614d700 1 --2- 172.21.15.3:0/3555627116 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3de80385f0 0x7f3de803aab0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:38.518 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.196+0000 7f3e0614d700 1 --2- 172.21.15.3:0/3555627116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e00106a40 0x7f3e00199d10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:38.518 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.196+0000 7f3e0614d700 1 -- 172.21.15.3:0/3555627116 >> 172.21.15.3:0/3555627116 conn(0x7f3e00100490 msgr2=0x7f3e00101b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:38.518 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.196+0000 7f3e0614d700 1 -- 172.21.15.3:0/3555627116 shutdown_connections 2023-07-20T01:36:38.519 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:38.196+0000 7f3e0614d700 1 -- 172.21.15.3:0/3555627116 wait complete. 2023-07-20T01:36:38.519 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial admin user... 2023-07-20T01:36:38.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:38 smithi003 ceph-mon[96422]: [20/Jul/2023:01:36:37] ENGINE Bus STARTING 2023-07-20T01:36:38.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:38 smithi003 ceph-mon[96422]: [20/Jul/2023:01:36:37] ENGINE Serving on https://172.21.15.3:7150 2023-07-20T01:36:38.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:38 smithi003 ceph-mon[96422]: [20/Jul/2023:01:36:37] ENGINE Serving on http://172.21.15.3:8765 2023-07-20T01:36:38.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:38 smithi003 ceph-mon[96422]: [20/Jul/2023:01:36:37] ENGINE Bus STARTED 2023-07-20T01:36:38.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:38.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:38.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:39.843 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:39 smithi003 ceph-mon[96422]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:36:39.843 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:39 smithi003 ceph-mon[96422]: mgrmap e12: x(active, since 2s) 2023-07-20T01:36:39.843 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:39 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:40.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$P/lU1SWbTV/Km0Zbc3h2BeLM..xPzUzVcrQUF9NwC7YxEg48bUP8W", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689816999, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-20T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.024+0000 7f01d4b58700 1 Processor -- start 2023-07-20T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.025+0000 7f01d4b58700 1 -- start start 2023-07-20T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.025+0000 7f01d4b58700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0104a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.025+0000 7f01d4b58700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0105060 con 0x7f01d0104670 2023-07-20T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.025+0000 7f01ce59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0104a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.025+0000 7f01ce59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0104a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41756/0 (socket says 172.21.15.3:41756) 2023-07-20T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.025+0000 7f01ce59c700 1 -- 172.21.15.3:0/2021109533 learned_addr learned my addr 172.21.15.3:0/2021109533 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:40.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.025+0000 7f01ce59c700 1 -- 172.21.15.3:0/2021109533 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d0105870 con 0x7f01d0104670 2023-07-20T01:36:40.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.026+0000 7f01ce59c700 1 --2- 172.21.15.3:0/2021109533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0104a90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009e80 tx=0x7f01c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7363b57d60b36d7 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:40.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.026+0000 7f01cd59a700 1 -- 172.21.15.3:0/2021109533 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01c800e070 con 0x7f01d0104670 2023-07-20T01:36:40.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.026+0000 7f01cd59a700 1 -- 172.21.15.3:0/2021109533 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01c800b8c0 con 0x7f01d0104670 2023-07-20T01:36:40.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.026+0000 7f01cd59a700 1 -- 172.21.15.3:0/2021109533 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01c80043e0 con 0x7f01d0104670 2023-07-20T01:36:40.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.026+0000 7f01d4b58700 1 -- 172.21.15.3:0/2021109533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 msgr2=0x7f01d0104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:40.158 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.026+0000 7f01d4b58700 1 --2- 172.21.15.3:0/2021109533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0104a90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009e80 tx=0x7f01c800b330 comp rx=0 tx=0).stop 2023-07-20T01:36:40.159 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.026+0000 7f01d4b58700 1 -- 172.21.15.3:0/2021109533 shutdown_connections 2023-07-20T01:36:40.159 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.026+0000 7f01d4b58700 1 --2- 172.21.15.3:0/2021109533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0104a90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:40.159 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.026+0000 7f01d4b58700 1 -- 172.21.15.3:0/2021109533 >> 172.21.15.3:0/2021109533 conn(0x7f01d00ffc10 msgr2=0x7f01d0102050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:40.159 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.027+0000 7f01d4b58700 1 -- 172.21.15.3:0/2021109533 shutdown_connections 2023-07-20T01:36:40.159 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.027+0000 7f01d4b58700 1 -- 172.21.15.3:0/2021109533 wait complete. 2023-07-20T01:36:40.159 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.027+0000 7f01d4b58700 1 Processor -- start 2023-07-20T01:36:40.160 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.027+0000 7f01d4b58700 1 -- start start 2023-07-20T01:36:40.160 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01d4b58700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0190430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:40.160 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01d4b58700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0190970 con 0x7f01d0104670 2023-07-20T01:36:40.160 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01ce59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0190430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:40.160 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01ce59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0190430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41758/0 (socket says 172.21.15.3:41758) 2023-07-20T01:36:40.161 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01ce59c700 1 -- 172.21.15.3:0/730759069 learned_addr learned my addr 172.21.15.3:0/730759069 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:40.161 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01ce59c700 1 -- 172.21.15.3:0/730759069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c8009b30 con 0x7f01d0104670 2023-07-20T01:36:40.161 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01ce59c700 1 --2- 172.21.15.3:0/730759069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0190430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f01c8004070 tx=0x7f01c8003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:40.161 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01c77fe700 1 -- 172.21.15.3:0/730759069 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01c800e040 con 0x7f01d0104670 2023-07-20T01:36:40.161 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01c77fe700 1 -- 172.21.15.3:0/730759069 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01c800bdf0 con 0x7f01d0104670 2023-07-20T01:36:40.161 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01d4b58700 1 -- 172.21.15.3:0/730759069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d018ea80 con 0x7f01d0104670 2023-07-20T01:36:40.162 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01c77fe700 1 -- 172.21.15.3:0/730759069 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01c801c750 con 0x7f01d0104670 2023-07-20T01:36:40.162 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.028+0000 7f01d4b58700 1 -- 172.21.15.3:0/730759069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d018ef00 con 0x7f01d0104670 2023-07-20T01:36:40.162 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.029+0000 7f01c77fe700 1 -- 172.21.15.3:0/730759069 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f01c801c8b0 con 0x7f01d0104670 2023-07-20T01:36:40.162 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.029+0000 7f01c77fe700 1 --2- 172.21.15.3:0/730759069 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01bc0387b0 0x7f01bc03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:40.162 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.029+0000 7f01c77fe700 1 -- 172.21.15.3:0/730759069 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f01c8012920 con 0x7f01d0104670 2023-07-20T01:36:40.162 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.029+0000 7f01cdd9b700 1 --2- 172.21.15.3:0/730759069 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01bc0387b0 0x7f01bc03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.030+0000 7f01c57fa700 1 -- 172.21.15.3:0/730759069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d0061da0 con 0x7f01d0104670 2023-07-20T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.030+0000 7f01cdd9b700 1 --2- 172.21.15.3:0/730759069 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01bc0387b0 0x7f01bc03ac70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01b8006fd0 tx=0x7f01b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.034+0000 7f01c77fe700 1 -- 172.21.15.3:0/730759069 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f01c8017070 con 0x7f01d0104670 2023-07-20T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.436+0000 7f01c57fa700 1 -- 172.21.15.3:0/730759069 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f01d0061bc0 con 0x7f01bc0387b0 2023-07-20T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.683+0000 7f01c77fe700 1 -- 172.21.15.3:0/730759069 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f01d0061bc0 con 0x7f01bc0387b0 2023-07-20T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.685+0000 7f01c57fa700 1 -- 172.21.15.3:0/730759069 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01bc0387b0 msgr2=0x7f01bc03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.685+0000 7f01c57fa700 1 --2- 172.21.15.3:0/730759069 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01bc0387b0 0x7f01bc03ac70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01b8006fd0 tx=0x7f01b8006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:40.164 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.685+0000 7f01c57fa700 1 -- 172.21.15.3:0/730759069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 msgr2=0x7f01d0190430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:40.164 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.685+0000 7f01c57fa700 1 --2- 172.21.15.3:0/730759069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0190430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f01c8004070 tx=0x7f01c8003d00 comp rx=0 tx=0).stop 2023-07-20T01:36:40.164 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.685+0000 7f01c57fa700 1 -- 172.21.15.3:0/730759069 shutdown_connections 2023-07-20T01:36:40.164 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.685+0000 7f01c57fa700 1 --2- 172.21.15.3:0/730759069 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01bc0387b0 0x7f01bc03ac70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:40.164 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.685+0000 7f01c57fa700 1 --2- 172.21.15.3:0/730759069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01d0104670 0x7f01d0190430 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:40.164 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.685+0000 7f01c57fa700 1 -- 172.21.15.3:0/730759069 >> 172.21.15.3:0/730759069 conn(0x7f01d00ffc10 msgr2=0x7f01d0109430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:40.165 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.685+0000 7f01c57fa700 1 -- 172.21.15.3:0/730759069 shutdown_connections 2023-07-20T01:36:40.165 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:39.685+0000 7f01c57fa700 1 -- 172.21.15.3:0/730759069 wait complete. 2023-07-20T01:36:40.165 INFO:teuthology.orchestra.run.smithi003.stdout:Fetching dashboard port number... 2023-07-20T01:36:40.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:40 smithi003 ceph-mon[96422]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:36:40.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:40 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:41.493 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 8443 2023-07-20T01:36:41.494 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.783+0000 7f30d8e33700 1 Processor -- start 2023-07-20T01:36:41.494 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.783+0000 7f30d8e33700 1 -- start start 2023-07-20T01:36:41.494 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.783+0000 7f30d8e33700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d4104cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:41.494 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.783+0000 7f30d8e33700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d4073f30 con 0x7f30d41048b0 2023-07-20T01:36:41.494 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.783+0000 7f30d259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d4104cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:41.494 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.783+0000 7f30d259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d4104cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41762/0 (socket says 172.21.15.3:41762) 2023-07-20T01:36:41.494 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.783+0000 7f30d259c700 1 -- 172.21.15.3:0/1946504524 learned_addr learned my addr 172.21.15.3:0/1946504524 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:41.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.784+0000 7f30d259c700 1 -- 172.21.15.3:0/1946504524 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d4105210 con 0x7f30d41048b0 2023-07-20T01:36:41.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.784+0000 7f30d259c700 1 --2- 172.21.15.3:0/1946504524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d4104cd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f30c40098a0 tx=0x7f30c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f2de9c5421284 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:41.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.784+0000 7f30d159a700 1 -- 172.21.15.3:0/1946504524 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f30c4004160 con 0x7f30d41048b0 2023-07-20T01:36:41.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.784+0000 7f30d159a700 1 -- 172.21.15.3:0/1946504524 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30c40042c0 con 0x7f30d41048b0 2023-07-20T01:36:41.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.784+0000 7f30d159a700 1 -- 172.21.15.3:0/1946504524 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f30c40119b0 con 0x7f30d41048b0 2023-07-20T01:36:41.495 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.784+0000 7f30d8e33700 1 -- 172.21.15.3:0/1946504524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 msgr2=0x7f30d4104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:41.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.784+0000 7f30d8e33700 1 --2- 172.21.15.3:0/1946504524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d4104cd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f30c40098a0 tx=0x7f30c4009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:41.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.785+0000 7f30d8e33700 1 -- 172.21.15.3:0/1946504524 shutdown_connections 2023-07-20T01:36:41.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.785+0000 7f30d8e33700 1 --2- 172.21.15.3:0/1946504524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d4104cd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:41.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.785+0000 7f30d8e33700 1 -- 172.21.15.3:0/1946504524 >> 172.21.15.3:0/1946504524 conn(0x7f30d4100490 msgr2=0x7f30d41028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:41.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.785+0000 7f30d8e33700 1 -- 172.21.15.3:0/1946504524 shutdown_connections 2023-07-20T01:36:41.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.785+0000 7f30d8e33700 1 -- 172.21.15.3:0/1946504524 wait complete. 2023-07-20T01:36:41.496 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.785+0000 7f30d8e33700 1 Processor -- start 2023-07-20T01:36:41.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.786+0000 7f30d8e33700 1 -- start start 2023-07-20T01:36:41.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.786+0000 7f30d8e33700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d419bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:41.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.786+0000 7f30d8e33700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d419c430 con 0x7f30d41048b0 2023-07-20T01:36:41.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.786+0000 7f30d259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d419bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:41.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.786+0000 7f30d259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d419bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41764/0 (socket says 172.21.15.3:41764) 2023-07-20T01:36:41.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.786+0000 7f30d259c700 1 -- 172.21.15.3:0/1550209460 learned_addr learned my addr 172.21.15.3:0/1550209460 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:41.497 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.786+0000 7f30d259c700 1 -- 172.21.15.3:0/1550209460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c4009550 con 0x7f30d41048b0 2023-07-20T01:36:41.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.786+0000 7f30d259c700 1 --2- 172.21.15.3:0/1550209460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d419bef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f30c4011980 tx=0x7f30c4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:41.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.786+0000 7f30c37fe700 1 -- 172.21.15.3:0/1550209460 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f30c4003a60 con 0x7f30d41048b0 2023-07-20T01:36:41.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.786+0000 7f30d8e33700 1 -- 172.21.15.3:0/1550209460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d419c630 con 0x7f30d41048b0 2023-07-20T01:36:41.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.786+0000 7f30c37fe700 1 -- 172.21.15.3:0/1550209460 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30c4003bc0 con 0x7f30d41048b0 2023-07-20T01:36:41.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.787+0000 7f30c37fe700 1 -- 172.21.15.3:0/1550209460 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f30c401a670 con 0x7f30d41048b0 2023-07-20T01:36:41.498 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.787+0000 7f30d8e33700 1 -- 172.21.15.3:0/1550209460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d419ca50 con 0x7f30d41048b0 2023-07-20T01:36:41.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.788+0000 7f30c37fe700 1 -- 172.21.15.3:0/1550209460 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f30c4024a00 con 0x7f30d41048b0 2023-07-20T01:36:41.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.788+0000 7f30d8e33700 1 -- 172.21.15.3:0/1550209460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d4195770 con 0x7f30d41048b0 2023-07-20T01:36:41.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.788+0000 7f30c37fe700 1 --2- 172.21.15.3:0/1550209460 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f30bc038760 0x7f30bc03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:41.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.788+0000 7f30c37fe700 1 -- 172.21.15.3:0/1550209460 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f30c404ca50 con 0x7f30d41048b0 2023-07-20T01:36:41.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.788+0000 7f30d1d9b700 1 --2- 172.21.15.3:0/1550209460 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f30bc038760 0x7f30bc03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:41.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.788+0000 7f30d1d9b700 1 --2- 172.21.15.3:0/1550209460 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f30bc038760 0x7f30bc03ac20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f30c8006fd0 tx=0x7f30c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:41.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:40.793+0000 7f30c37fe700 1 -- 172.21.15.3:0/1550209460 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f30c40189a0 con 0x7f30d41048b0 2023-07-20T01:36:41.499 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.150+0000 7f30d8e33700 1 -- 172.21.15.3:0/1550209460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f30d402ce00 con 0x7f30d41048b0 2023-07-20T01:36:41.500 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.151+0000 7f30c37fe700 1 -- 172.21.15.3:0/1550209460 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f30c4018350 con 0x7f30d41048b0 2023-07-20T01:36:41.500 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.153+0000 7f30d8e33700 1 -- 172.21.15.3:0/1550209460 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f30bc038760 msgr2=0x7f30bc03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:41.500 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.153+0000 7f30d8e33700 1 --2- 172.21.15.3:0/1550209460 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f30bc038760 0x7f30bc03ac20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f30c8006fd0 tx=0x7f30c8006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:41.500 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.153+0000 7f30d8e33700 1 -- 172.21.15.3:0/1550209460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 msgr2=0x7f30d419bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:41.500 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.153+0000 7f30d8e33700 1 --2- 172.21.15.3:0/1550209460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d419bef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f30c4011980 tx=0x7f30c4011a60 comp rx=0 tx=0).stop 2023-07-20T01:36:41.500 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.153+0000 7f30d8e33700 1 -- 172.21.15.3:0/1550209460 shutdown_connections 2023-07-20T01:36:41.501 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.153+0000 7f30d8e33700 1 --2- 172.21.15.3:0/1550209460 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f30bc038760 0x7f30bc03ac20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:41.501 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.153+0000 7f30d8e33700 1 --2- 172.21.15.3:0/1550209460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30d41048b0 0x7f30d419bef0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:41.501 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.153+0000 7f30d8e33700 1 -- 172.21.15.3:0/1550209460 >> 172.21.15.3:0/1550209460 conn(0x7f30d4100490 msgr2=0x7f30d4190470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:41.501 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.153+0000 7f30d8e33700 1 -- 172.21.15.3:0/1550209460 shutdown_connections 2023-07-20T01:36:41.501 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:41.153+0000 7f30d8e33700 1 -- 172.21.15.3:0/1550209460 wait complete. 2023-07-20T01:36:41.502 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T01:36:41.502 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2023-07-20T01:36:41.509 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T01:36:41.509 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-07-20T01:36:41.509 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2023-07-20T01:36:41.510 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-20T01:36:41.510 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph Dashboard is now available at: 2023-07-20T01:36:41.511 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:36:41.511 INFO:teuthology.orchestra.run.smithi003.stdout: URL: https://smithi003:8443/ 2023-07-20T01:36:41.511 INFO:teuthology.orchestra.run.smithi003.stdout: User: admin 2023-07-20T01:36:41.511 INFO:teuthology.orchestra.run.smithi003.stdout: Password: nb030f6yf5 2023-07-20T01:36:41.511 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:36:41.512 INFO:teuthology.orchestra.run.smithi003.stdout:Saving cluster configuration to /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config directory 2023-07-20T01:36:41.512 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling autotune for osd_memory_target 2023-07-20T01:36:41.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:41 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/1550209460' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-20T01:36:42.653 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.025+0000 7f8a6ed21700 1 Processor -- start 2023-07-20T01:36:42.654 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.025+0000 7f8a6ed21700 1 -- start start 2023-07-20T01:36:42.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.025+0000 7f8a6ed21700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a68108eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:42.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.025+0000 7f8a6ed21700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a68079dd0 con 0x7f8a68106ac0 2023-07-20T01:36:42.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.026+0000 7f8a6cabd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a68108eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:42.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.026+0000 7f8a6cabd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a68108eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41772/0 (socket says 172.21.15.3:41772) 2023-07-20T01:36:42.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.026+0000 7f8a6cabd700 1 -- 172.21.15.3:0/977938772 learned_addr learned my addr 172.21.15.3:0/977938772 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:42.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.026+0000 7f8a6cabd700 1 -- 172.21.15.3:0/977938772 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a681093f0 con 0x7f8a68106ac0 2023-07-20T01:36:42.655 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.026+0000 7f8a6cabd700 1 --2- 172.21.15.3:0/977938772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a68108eb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0098a0 tx=0x7f8a5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2dc09b42c949e46 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:42.656 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.026+0000 7f8a677fe700 1 -- 172.21.15.3:0/977938772 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a5c004160 con 0x7f8a68106ac0 2023-07-20T01:36:42.659 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.026+0000 7f8a677fe700 1 -- 172.21.15.3:0/977938772 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a5c0042c0 con 0x7f8a68106ac0 2023-07-20T01:36:42.659 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.026+0000 7f8a677fe700 1 -- 172.21.15.3:0/977938772 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a5c0119b0 con 0x7f8a68106ac0 2023-07-20T01:36:42.659 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.027+0000 7f8a6ed21700 1 -- 172.21.15.3:0/977938772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 msgr2=0x7f8a68108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:42.659 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.027+0000 7f8a6ed21700 1 --2- 172.21.15.3:0/977938772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a68108eb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0098a0 tx=0x7f8a5c009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:42.659 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.027+0000 7f8a6ed21700 1 -- 172.21.15.3:0/977938772 shutdown_connections 2023-07-20T01:36:42.659 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.027+0000 7f8a6ed21700 1 --2- 172.21.15.3:0/977938772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a68108eb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:42.660 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.027+0000 7f8a6ed21700 1 -- 172.21.15.3:0/977938772 >> 172.21.15.3:0/977938772 conn(0x7f8a681004d0 msgr2=0x7f8a681028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:42.660 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.027+0000 7f8a6ed21700 1 -- 172.21.15.3:0/977938772 shutdown_connections 2023-07-20T01:36:42.660 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.027+0000 7f8a6ed21700 1 -- 172.21.15.3:0/977938772 wait complete. 2023-07-20T01:36:42.660 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.028+0000 7f8a6ed21700 1 Processor -- start 2023-07-20T01:36:42.660 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.028+0000 7f8a6ed21700 1 -- start start 2023-07-20T01:36:42.660 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.028+0000 7f8a6ed21700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a6819bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:42.660 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.028+0000 7f8a6ed21700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6819c4d0 con 0x7f8a68106ac0 2023-07-20T01:36:42.660 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.028+0000 7f8a6cabd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a6819bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:42.661 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.028+0000 7f8a6cabd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a6819bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41774/0 (socket says 172.21.15.3:41774) 2023-07-20T01:36:42.661 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.028+0000 7f8a6cabd700 1 -- 172.21.15.3:0/747371388 learned_addr learned my addr 172.21.15.3:0/747371388 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:42.661 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.028+0000 7f8a6cabd700 1 -- 172.21.15.3:0/747371388 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c009550 con 0x7f8a68106ac0 2023-07-20T01:36:42.661 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.028+0000 7f8a6cabd700 1 --2- 172.21.15.3:0/747371388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a6819bf90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c006af0 tx=0x7f8a5c011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:42.661 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.029+0000 7f8a65ffb700 1 -- 172.21.15.3:0/747371388 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a5c003a40 con 0x7f8a68106ac0 2023-07-20T01:36:42.661 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.029+0000 7f8a65ffb700 1 -- 172.21.15.3:0/747371388 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a5c003ba0 con 0x7f8a68106ac0 2023-07-20T01:36:42.661 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.029+0000 7f8a6ed21700 1 -- 172.21.15.3:0/747371388 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6819c6d0 con 0x7f8a68106ac0 2023-07-20T01:36:42.661 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.029+0000 7f8a65ffb700 1 -- 172.21.15.3:0/747371388 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a5c0196c0 con 0x7f8a68106ac0 2023-07-20T01:36:42.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.029+0000 7f8a6ed21700 1 -- 172.21.15.3:0/747371388 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6819caf0 con 0x7f8a68106ac0 2023-07-20T01:36:42.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.030+0000 7f8a6ed21700 1 -- 172.21.15.3:0/747371388 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a68196220 con 0x7f8a68106ac0 2023-07-20T01:36:42.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.030+0000 7f8a65ffb700 1 -- 172.21.15.3:0/747371388 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f8a5c01aa80 con 0x7f8a68106ac0 2023-07-20T01:36:42.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.030+0000 7f8a65ffb700 1 --2- 172.21.15.3:0/747371388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8a58038760 0x7f8a5803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:42.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.031+0000 7f8a65ffb700 1 -- 172.21.15.3:0/747371388 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8a5c04cd20 con 0x7f8a68106ac0 2023-07-20T01:36:42.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.031+0000 7f8a67fff700 1 --2- 172.21.15.3:0/747371388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8a58038760 0x7f8a5803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:42.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.031+0000 7f8a67fff700 1 --2- 172.21.15.3:0/747371388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8a58038760 0x7f8a5803ac20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8a54006fd0 tx=0x7f8a54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:42.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.035+0000 7f8a65ffb700 1 -- 172.21.15.3:0/747371388 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8a5c01a430 con 0x7f8a68106ac0 2023-07-20T01:36:42.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.392+0000 7f8a6ed21700 1 -- 172.21.15.3:0/747371388 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f8a68196220 con 0x7f8a68106ac0 2023-07-20T01:36:42.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.393+0000 7f8a65ffb700 1 -- 172.21.15.3:0/747371388 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f8a5c023390 con 0x7f8a68106ac0 2023-07-20T01:36:42.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.395+0000 7f8a6ed21700 1 -- 172.21.15.3:0/747371388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8a58038760 msgr2=0x7f8a5803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:42.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.395+0000 7f8a6ed21700 1 --2- 172.21.15.3:0/747371388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8a58038760 0x7f8a5803ac20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8a54006fd0 tx=0x7f8a54006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:42.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.395+0000 7f8a6ed21700 1 -- 172.21.15.3:0/747371388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 msgr2=0x7f8a6819bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:42.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.395+0000 7f8a6ed21700 1 --2- 172.21.15.3:0/747371388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a6819bf90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c006af0 tx=0x7f8a5c011a10 comp rx=0 tx=0).stop 2023-07-20T01:36:42.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.395+0000 7f8a6ed21700 1 -- 172.21.15.3:0/747371388 shutdown_connections 2023-07-20T01:36:42.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.395+0000 7f8a6ed21700 1 --2- 172.21.15.3:0/747371388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8a58038760 0x7f8a5803ac20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:42.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.395+0000 7f8a6ed21700 1 --2- 172.21.15.3:0/747371388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a68106ac0 0x7f8a6819bf90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:42.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.395+0000 7f8a6ed21700 1 -- 172.21.15.3:0/747371388 >> 172.21.15.3:0/747371388 conn(0x7f8a681004d0 msgr2=0x7f8a68101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:42.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.395+0000 7f8a6ed21700 1 -- 172.21.15.3:0/747371388 shutdown_connections 2023-07-20T01:36:42.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:42.395+0000 7f8a6ed21700 1 -- 172.21.15.3:0/747371388 wait complete. 2023-07-20T01:36:43.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:43 smithi003 ceph-mon[96422]: mgrmap e13: x(active, since 6s) 2023-07-20T01:36:44.105 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8803f1a700 1 Processor -- start 2023-07-20T01:36:44.105 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8803f1a700 1 -- start start 2023-07-20T01:36:44.105 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8803f1a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc104cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:44.105 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8803f1a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc073f30 con 0x7f87fc1048b0 2023-07-20T01:36:44.105 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8801cb6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc104cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:44.106 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8801cb6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc104cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41778/0 (socket says 172.21.15.3:41778) 2023-07-20T01:36:44.106 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8801cb6700 1 -- 172.21.15.3:0/2024804565 learned_addr learned my addr 172.21.15.3:0/2024804565 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:44.106 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8801cb6700 1 -- 172.21.15.3:0/2024804565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc105210 con 0x7f87fc1048b0 2023-07-20T01:36:44.106 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8801cb6700 1 --2- 172.21.15.3:0/2024804565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc104cd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f87f80098a0 tx=0x7f87f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b46f819b37b7056 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:44.106 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8800cb4700 1 -- 172.21.15.3:0/2024804565 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f8004160 con 0x7f87fc1048b0 2023-07-20T01:36:44.106 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8800cb4700 1 -- 172.21.15.3:0/2024804565 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87f80042c0 con 0x7f87fc1048b0 2023-07-20T01:36:44.106 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.136+0000 7f8800cb4700 1 -- 172.21.15.3:0/2024804565 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f80119b0 con 0x7f87fc1048b0 2023-07-20T01:36:44.106 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.137+0000 7f8803f1a700 1 -- 172.21.15.3:0/2024804565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 msgr2=0x7f87fc104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:44.107 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.137+0000 7f8803f1a700 1 --2- 172.21.15.3:0/2024804565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc104cd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f87f80098a0 tx=0x7f87f8009bb0 comp rx=0 tx=0).stop 2023-07-20T01:36:44.107 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.137+0000 7f8803f1a700 1 -- 172.21.15.3:0/2024804565 shutdown_connections 2023-07-20T01:36:44.107 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.137+0000 7f8803f1a700 1 --2- 172.21.15.3:0/2024804565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc104cd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:44.107 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.137+0000 7f8803f1a700 1 -- 172.21.15.3:0/2024804565 >> 172.21.15.3:0/2024804565 conn(0x7f87fc100490 msgr2=0x7f87fc1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:44.107 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.137+0000 7f8803f1a700 1 -- 172.21.15.3:0/2024804565 shutdown_connections 2023-07-20T01:36:44.108 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.137+0000 7f8803f1a700 1 -- 172.21.15.3:0/2024804565 wait complete. 2023-07-20T01:36:44.108 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.138+0000 7f8803f1a700 1 Processor -- start 2023-07-20T01:36:44.108 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.138+0000 7f8803f1a700 1 -- start start 2023-07-20T01:36:44.108 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.138+0000 7f8803f1a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc19bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:44.108 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.138+0000 7f8803f1a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc19c430 con 0x7f87fc1048b0 2023-07-20T01:36:44.108 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.138+0000 7f8801cb6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc19bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:44.109 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.138+0000 7f8801cb6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc19bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41780/0 (socket says 172.21.15.3:41780) 2023-07-20T01:36:44.109 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.138+0000 7f8801cb6700 1 -- 172.21.15.3:0/970626391 learned_addr learned my addr 172.21.15.3:0/970626391 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:44.109 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.138+0000 7f8801cb6700 1 -- 172.21.15.3:0/970626391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f8009550 con 0x7f87fc1048b0 2023-07-20T01:36:44.109 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.139+0000 7f8801cb6700 1 --2- 172.21.15.3:0/970626391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc19bef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f87f8003e60 tx=0x7f87f8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:44.109 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.139+0000 7f87f2ffd700 1 -- 172.21.15.3:0/970626391 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f8003970 con 0x7f87fc1048b0 2023-07-20T01:36:44.109 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.139+0000 7f8803f1a700 1 -- 172.21.15.3:0/970626391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc19c630 con 0x7f87fc1048b0 2023-07-20T01:36:44.109 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.139+0000 7f87f2ffd700 1 -- 172.21.15.3:0/970626391 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87f8003ad0 con 0x7f87fc1048b0 2023-07-20T01:36:44.109 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.139+0000 7f87f2ffd700 1 -- 172.21.15.3:0/970626391 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f80195e0 con 0x7f87fc1048b0 2023-07-20T01:36:44.110 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.139+0000 7f8803f1a700 1 -- 172.21.15.3:0/970626391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc19ca50 con 0x7f87fc1048b0 2023-07-20T01:36:44.110 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.140+0000 7f87f2ffd700 1 -- 172.21.15.3:0/970626391 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f87f80198e0 con 0x7f87fc1048b0 2023-07-20T01:36:44.110 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.140+0000 7f87f2ffd700 1 --2- 172.21.15.3:0/970626391 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f87e80387b0 0x7f87e803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:44.110 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.140+0000 7f87f2ffd700 1 -- 172.21.15.3:0/970626391 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f87f804cce0 con 0x7f87fc1048b0 2023-07-20T01:36:44.110 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.140+0000 7f8803f1a700 1 -- 172.21.15.3:0/970626391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc195770 con 0x7f87fc1048b0 2023-07-20T01:36:44.110 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.140+0000 7f88014b5700 1 --2- 172.21.15.3:0/970626391 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f87e80387b0 0x7f87e803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:44.110 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.141+0000 7f88014b5700 1 --2- 172.21.15.3:0/970626391 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f87e80387b0 0x7f87e803ac70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87ec006fd0 tx=0x7f87ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:44.111 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.146+0000 7f87f2ffd700 1 -- 172.21.15.3:0/970626391 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f87f8019c30 con 0x7f87fc1048b0 2023-07-20T01:36:44.111 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.713+0000 7f8803f1a700 1 -- 172.21.15.3:0/970626391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f87fc061da0 con 0x7f87fc1048b0 2023-07-20T01:36:44.111 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.716+0000 7f87f2ffd700 1 -- 172.21.15.3:0/970626391 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f87f8023b40 con 0x7f87fc1048b0 2023-07-20T01:36:44.111 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.718+0000 7f8803f1a700 1 -- 172.21.15.3:0/970626391 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f87e80387b0 msgr2=0x7f87e803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:44.111 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.718+0000 7f8803f1a700 1 --2- 172.21.15.3:0/970626391 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f87e80387b0 0x7f87e803ac70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87ec006fd0 tx=0x7f87ec006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:44.111 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.718+0000 7f8803f1a700 1 -- 172.21.15.3:0/970626391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 msgr2=0x7f87fc19bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:44.111 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.718+0000 7f8803f1a700 1 --2- 172.21.15.3:0/970626391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc19bef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f87f8003e60 tx=0x7f87f8011aa0 comp rx=0 tx=0).stop 2023-07-20T01:36:44.111 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.718+0000 7f8803f1a700 1 -- 172.21.15.3:0/970626391 shutdown_connections 2023-07-20T01:36:44.112 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.718+0000 7f8803f1a700 1 --2- 172.21.15.3:0/970626391 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f87e80387b0 0x7f87e803ac70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:44.112 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.718+0000 7f8803f1a700 1 --2- 172.21.15.3:0/970626391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87fc1048b0 0x7f87fc19bef0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:44.112 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.718+0000 7f8803f1a700 1 -- 172.21.15.3:0/970626391 >> 172.21.15.3:0/970626391 conn(0x7f87fc100490 msgr2=0x7f87fc190470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:44.112 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.718+0000 7f8803f1a700 1 -- 172.21.15.3:0/970626391 shutdown_connections 2023-07-20T01:36:44.112 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2023-07-20T01:36:43.718+0000 7f8803f1a700 1 -- 172.21.15.3:0/970626391 wait complete. 2023-07-20T01:36:44.112 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-20T01:36:44.112 INFO:teuthology.orchestra.run.smithi003.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-20T01:36:44.112 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:36:44.113 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:36:44.113 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:36:44.113 INFO:teuthology.orchestra.run.smithi003.stdout:Or, if you are only running a single cluster on this host: 2023-07-20T01:36:44.113 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:36:44.113 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-20T01:36:44.113 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:36:44.113 INFO:teuthology.orchestra.run.smithi003.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-20T01:36:44.114 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:36:44.114 INFO:teuthology.orchestra.run.smithi003.stdout: ceph telemetry on 2023-07-20T01:36:44.114 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:36:44.114 INFO:teuthology.orchestra.run.smithi003.stdout:For more information see: 2023-07-20T01:36:44.114 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:36:44.114 INFO:teuthology.orchestra.run.smithi003.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-20T01:36:44.114 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:36:44.114 INFO:teuthology.orchestra.run.smithi003.stdout:Bootstrap complete. 2023-07-20T01:36:44.142 INFO:tasks.cephadm:Fetching config... 2023-07-20T01:36:44.143 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:36:44.144 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-20T01:36:44.160 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-20T01:36:44.161 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:36:44.161 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-20T01:36:44.219 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-20T01:36:44.219 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:36:44.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-20T01:36:44.287 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-20T01:36:44.287 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:36:44.288 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-20T01:36:44.344 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-20T01:36:44.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdq/XYHDg/SpMUlL9fgndxVP3LLufcPhBfdORMhq0puHfPWSwQTzoOnbnyUdZQOiXssoMIprxef9rwd1SPLxtZOwxqos4chnOvcwoPg228YRjh4p6xAOY8X9bHJFcNVX+gvPZGYFipnjO6Svl/i2PJwi4AwG9wUFmTIYkFq+dOA0t/x/zMKqTVCj5jziEkQrZp73bad14r3kFjkFBPMyjbB+rGiUKfTRcv2J6EjVyxjMnwlknEXGy3pdbKhGLSBz3FpneAw9nPFZX+9YJC8x4k/U1BmFmWhX7B0KxmYhN9r6ei3jFhCx2ytcDrin76Inxz/8iMAEMR5BoHy5//ajElTsSUDyhHZDyQ4oTrmrG8z0J97GPz8DzMC4LAzgrwtYpnFDYhtP3kn5jHPGOL6X8DuY2oKr8AISIUvznFy8t2XudOvh2DB/tFbURFu8DWx1wWGZnKOIqid6qp3prJWvOPGIOVwsxPiFe36ZQdRZHX5yh0t4dDKbMiOV0rzvu861U= ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T01:36:44.424 INFO:teuthology.orchestra.run.smithi003.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdq/XYHDg/SpMUlL9fgndxVP3LLufcPhBfdORMhq0puHfPWSwQTzoOnbnyUdZQOiXssoMIprxef9rwd1SPLxtZOwxqos4chnOvcwoPg228YRjh4p6xAOY8X9bHJFcNVX+gvPZGYFipnjO6Svl/i2PJwi4AwG9wUFmTIYkFq+dOA0t/x/zMKqTVCj5jziEkQrZp73bad14r3kFjkFBPMyjbB+rGiUKfTRcv2J6EjVyxjMnwlknEXGy3pdbKhGLSBz3FpneAw9nPFZX+9YJC8x4k/U1BmFmWhX7B0KxmYhN9r6ei3jFhCx2ytcDrin76Inxz/8iMAEMR5BoHy5//ajElTsSUDyhHZDyQ4oTrmrG8z0J97GPz8DzMC4LAzgrwtYpnFDYhtP3kn5jHPGOL6X8DuY2oKr8AISIUvznFy8t2XudOvh2DB/tFbURFu8DWx1wWGZnKOIqid6qp3prJWvOPGIOVwsxPiFe36ZQdRZHX5yh0t4dDKbMiOV0rzvu861U= ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:36:44.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdq/XYHDg/SpMUlL9fgndxVP3LLufcPhBfdORMhq0puHfPWSwQTzoOnbnyUdZQOiXssoMIprxef9rwd1SPLxtZOwxqos4chnOvcwoPg228YRjh4p6xAOY8X9bHJFcNVX+gvPZGYFipnjO6Svl/i2PJwi4AwG9wUFmTIYkFq+dOA0t/x/zMKqTVCj5jziEkQrZp73bad14r3kFjkFBPMyjbB+rGiUKfTRcv2J6EjVyxjMnwlknEXGy3pdbKhGLSBz3FpneAw9nPFZX+9YJC8x4k/U1BmFmWhX7B0KxmYhN9r6ei3jFhCx2ytcDrin76Inxz/8iMAEMR5BoHy5//ajElTsSUDyhHZDyQ4oTrmrG8z0J97GPz8DzMC4LAzgrwtYpnFDYhtP3kn5jHPGOL6X8DuY2oKr8AISIUvznFy8t2XudOvh2DB/tFbURFu8DWx1wWGZnKOIqid6qp3prJWvOPGIOVwsxPiFe36ZQdRZHX5yh0t4dDKbMiOV0rzvu861U= ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T01:36:44.484 INFO:teuthology.orchestra.run.smithi043.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdq/XYHDg/SpMUlL9fgndxVP3LLufcPhBfdORMhq0puHfPWSwQTzoOnbnyUdZQOiXssoMIprxef9rwd1SPLxtZOwxqos4chnOvcwoPg228YRjh4p6xAOY8X9bHJFcNVX+gvPZGYFipnjO6Svl/i2PJwi4AwG9wUFmTIYkFq+dOA0t/x/zMKqTVCj5jziEkQrZp73bad14r3kFjkFBPMyjbB+rGiUKfTRcv2J6EjVyxjMnwlknEXGy3pdbKhGLSBz3FpneAw9nPFZX+9YJC8x4k/U1BmFmWhX7B0KxmYhN9r6ei3jFhCx2ytcDrin76Inxz/8iMAEMR5BoHy5//ajElTsSUDyhHZDyQ4oTrmrG8z0J97GPz8DzMC4LAzgrwtYpnFDYhtP3kn5jHPGOL6X8DuY2oKr8AISIUvznFy8t2XudOvh2DB/tFbURFu8DWx1wWGZnKOIqid6qp3prJWvOPGIOVwsxPiFe36ZQdRZHX5yh0t4dDKbMiOV0rzvu861U= ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:36:44.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdq/XYHDg/SpMUlL9fgndxVP3LLufcPhBfdORMhq0puHfPWSwQTzoOnbnyUdZQOiXssoMIprxef9rwd1SPLxtZOwxqos4chnOvcwoPg228YRjh4p6xAOY8X9bHJFcNVX+gvPZGYFipnjO6Svl/i2PJwi4AwG9wUFmTIYkFq+dOA0t/x/zMKqTVCj5jziEkQrZp73bad14r3kFjkFBPMyjbB+rGiUKfTRcv2J6EjVyxjMnwlknEXGy3pdbKhGLSBz3FpneAw9nPFZX+9YJC8x4k/U1BmFmWhX7B0KxmYhN9r6ei3jFhCx2ytcDrin76Inxz/8iMAEMR5BoHy5//ajElTsSUDyhHZDyQ4oTrmrG8z0J97GPz8DzMC4LAzgrwtYpnFDYhtP3kn5jHPGOL6X8DuY2oKr8AISIUvznFy8t2XudOvh2DB/tFbURFu8DWx1wWGZnKOIqid6qp3prJWvOPGIOVwsxPiFe36ZQdRZHX5yh0t4dDKbMiOV0rzvu861U= ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T01:36:44.544 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdq/XYHDg/SpMUlL9fgndxVP3LLufcPhBfdORMhq0puHfPWSwQTzoOnbnyUdZQOiXssoMIprxef9rwd1SPLxtZOwxqos4chnOvcwoPg228YRjh4p6xAOY8X9bHJFcNVX+gvPZGYFipnjO6Svl/i2PJwi4AwG9wUFmTIYkFq+dOA0t/x/zMKqTVCj5jziEkQrZp73bad14r3kFjkFBPMyjbB+rGiUKfTRcv2J6EjVyxjMnwlknEXGy3pdbKhGLSBz3FpneAw9nPFZX+9YJC8x4k/U1BmFmWhX7B0KxmYhN9r6ei3jFhCx2ytcDrin76Inxz/8iMAEMR5BoHy5//ajElTsSUDyhHZDyQ4oTrmrG8z0J97GPz8DzMC4LAzgrwtYpnFDYhtP3kn5jHPGOL6X8DuY2oKr8AISIUvznFy8t2XudOvh2DB/tFbURFu8DWx1wWGZnKOIqid6qp3prJWvOPGIOVwsxPiFe36ZQdRZHX5yh0t4dDKbMiOV0rzvu861U= ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:36:44.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-20T01:36:44.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:44 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/970626391' entity='client.admin' 2023-07-20T01:36:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:36:46.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.753+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1808686746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe00ff160 msgr2=0x7fcfe00ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:46.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.753+0000 7fcfe4d0c700 1 --2- 172.21.15.3:0/1808686746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe00ff160 0x7fcfe00ff580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009910 tx=0x7fcfd0009c20 comp rx=0 tx=0).stop 2023-07-20T01:36:46.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.753+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1808686746 shutdown_connections 2023-07-20T01:36:46.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.753+0000 7fcfe4d0c700 1 --2- 172.21.15.3:0/1808686746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe00ff160 0x7fcfe00ff580 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:46.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.753+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1808686746 >> 172.21.15.3:0/1808686746 conn(0x7fcfe00fa6e0 msgr2=0x7fcfe00fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:46.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.753+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1808686746 shutdown_connections 2023-07-20T01:36:46.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.753+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1808686746 wait complete. 2023-07-20T01:36:46.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.754+0000 7fcfe4d0c700 1 Processor -- start 2023-07-20T01:36:46.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.754+0000 7fcfe4d0c700 1 -- start start 2023-07-20T01:36:46.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.754+0000 7fcfe4d0c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe00ff160 0x7fcfe0068d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:46.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.754+0000 7fcfe4d0c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe01980c0 con 0x7fcfe00ff160 2023-07-20T01:36:46.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.754+0000 7fcfde59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe00ff160 0x7fcfe0068d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:46.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.754+0000 7fcfde59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe00ff160 0x7fcfe0068d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41786/0 (socket says 172.21.15.3:41786) 2023-07-20T01:36:46.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.754+0000 7fcfde59c700 1 -- 172.21.15.3:0/1991972101 learned_addr learned my addr 172.21.15.3:0/1991972101 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:46.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.754+0000 7fcfde59c700 1 -- 172.21.15.3:0/1991972101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd00095f0 con 0x7fcfe00ff160 2023-07-20T01:36:46.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.755+0000 7fcfde59c700 1 --2- 172.21.15.3:0/1991972101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe00ff160 0x7fcfe0068d60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0004f40 tx=0x7fcfd00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:46.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.755+0000 7fcfcf7fe700 1 -- 172.21.15.3:0/1991972101 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfd001d070 con 0x7fcfe00ff160 2023-07-20T01:36:46.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.755+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1991972101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe0198200 con 0x7fcfe00ff160 2023-07-20T01:36:46.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.755+0000 7fcfcf7fe700 1 -- 172.21.15.3:0/1991972101 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfd000bc10 con 0x7fcfe00ff160 2023-07-20T01:36:46.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.755+0000 7fcfcf7fe700 1 -- 172.21.15.3:0/1991972101 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfd0003a70 con 0x7fcfe00ff160 2023-07-20T01:36:46.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.755+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1991972101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe01986a0 con 0x7fcfe00ff160 2023-07-20T01:36:46.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.756+0000 7fcfcf7fe700 1 -- 172.21.15.3:0/1991972101 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fcfd0003d30 con 0x7fcfe00ff160 2023-07-20T01:36:46.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.756+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1991972101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe018dc70 con 0x7fcfe00ff160 2023-07-20T01:36:46.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.756+0000 7fcfcf7fe700 1 --2- 172.21.15.3:0/1991972101 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fcfc80387b0 0x7fcfc803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:46.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.756+0000 7fcfcf7fe700 1 -- 172.21.15.3:0/1991972101 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcfd004c510 con 0x7fcfe00ff160 2023-07-20T01:36:46.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.756+0000 7fcfddd9b700 1 --2- 172.21.15.3:0/1991972101 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fcfc80387b0 0x7fcfc803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:46.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.756+0000 7fcfddd9b700 1 --2- 172.21.15.3:0/1991972101 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fcfc80387b0 0x7fcfc803ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4006fd0 tx=0x7fcfd4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:46.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.761+0000 7fcfcf7fe700 1 -- 172.21.15.3:0/1991972101 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcfd0029770 con 0x7fcfe00ff160 2023-07-20T01:36:46.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.965+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1991972101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fcfe002c970 con 0x7fcfe00ff160 2023-07-20T01:36:46.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.968+0000 7fcfcf7fe700 1 -- 172.21.15.3:0/1991972101 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fcfd0026030 con 0x7fcfe00ff160 2023-07-20T01:36:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.970+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1991972101 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fcfc80387b0 msgr2=0x7fcfc803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.970+0000 7fcfe4d0c700 1 --2- 172.21.15.3:0/1991972101 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fcfc80387b0 0x7fcfc803ac70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4006fd0 tx=0x7fcfd4006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.970+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1991972101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe00ff160 msgr2=0x7fcfe0068d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.970+0000 7fcfe4d0c700 1 --2- 172.21.15.3:0/1991972101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe00ff160 0x7fcfe0068d60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0004f40 tx=0x7fcfd00057c0 comp rx=0 tx=0).stop 2023-07-20T01:36:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.971+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1991972101 shutdown_connections 2023-07-20T01:36:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.971+0000 7fcfe4d0c700 1 --2- 172.21.15.3:0/1991972101 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fcfc80387b0 0x7fcfc803ac70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.971+0000 7fcfe4d0c700 1 --2- 172.21.15.3:0/1991972101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe00ff160 0x7fcfe0068d60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.971+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1991972101 >> 172.21.15.3:0/1991972101 conn(0x7fcfe00fa6e0 msgr2=0x7fcfe00fb480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.971+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1991972101 shutdown_connections 2023-07-20T01:36:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:46.971+0000 7fcfe4d0c700 1 -- 172.21.15.3:0/1991972101 wait complete. 2023-07-20T01:36:47.464 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-20T01:36:47.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-20T01:36:48.024 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:36:48.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:47 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/1991972101' entity='client.admin' 2023-07-20T01:36:48.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:48.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:48.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi003", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:36:48.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:48.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:36:48.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:49.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.281+0000 7f4e03eaa700 1 -- 172.21.15.3:0/1125085117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dfc0ff160 msgr2=0x7f4dfc0ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:49.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.281+0000 7f4e03eaa700 1 --2- 172.21.15.3:0/1125085117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dfc0ff160 0x7f4dfc0ff580 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009910 tx=0x7f4df0009c20 comp rx=0 tx=0).stop 2023-07-20T01:36:49.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.281+0000 7f4e03eaa700 1 -- 172.21.15.3:0/1125085117 shutdown_connections 2023-07-20T01:36:49.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.281+0000 7f4e03eaa700 1 --2- 172.21.15.3:0/1125085117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dfc0ff160 0x7f4dfc0ff580 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:49.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.281+0000 7f4e03eaa700 1 -- 172.21.15.3:0/1125085117 >> 172.21.15.3:0/1125085117 conn(0x7f4dfc0fa6e0 msgr2=0x7f4dfc0fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:49.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.281+0000 7f4e03eaa700 1 -- 172.21.15.3:0/1125085117 shutdown_connections 2023-07-20T01:36:49.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.281+0000 7f4e03eaa700 1 -- 172.21.15.3:0/1125085117 wait complete. 2023-07-20T01:36:49.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.282+0000 7f4e03eaa700 1 Processor -- start 2023-07-20T01:36:49.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.282+0000 7f4e03eaa700 1 -- start start 2023-07-20T01:36:49.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.282+0000 7f4e03eaa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dfc0ff160 0x7f4dfc194270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:49.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.282+0000 7f4e03eaa700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc1947b0 con 0x7f4dfc0ff160 2023-07-20T01:36:49.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.282+0000 7f4e01c46700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dfc0ff160 0x7f4dfc194270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:49.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.282+0000 7f4e01c46700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dfc0ff160 0x7f4dfc194270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41792/0 (socket says 172.21.15.3:41792) 2023-07-20T01:36:49.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.282+0000 7f4e01c46700 1 -- 172.21.15.3:0/665644363 learned_addr learned my addr 172.21.15.3:0/665644363 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:49.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.283+0000 7f4e01c46700 1 -- 172.21.15.3:0/665644363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df00095f0 con 0x7f4dfc0ff160 2023-07-20T01:36:49.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.283+0000 7f4e01c46700 1 --2- 172.21.15.3:0/665644363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dfc0ff160 0x7f4dfc194270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4df0004f40 tx=0x7f4df00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:49.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.283+0000 7f4deeffd700 1 -- 172.21.15.3:0/665644363 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4df001d070 con 0x7f4dfc0ff160 2023-07-20T01:36:49.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.283+0000 7f4e03eaa700 1 -- 172.21.15.3:0/665644363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dfc1949b0 con 0x7f4dfc0ff160 2023-07-20T01:36:49.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.283+0000 7f4deeffd700 1 -- 172.21.15.3:0/665644363 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4df000bc10 con 0x7f4dfc0ff160 2023-07-20T01:36:49.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.283+0000 7f4deeffd700 1 -- 172.21.15.3:0/665644363 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4df0003a70 con 0x7f4dfc0ff160 2023-07-20T01:36:49.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.283+0000 7f4e03eaa700 1 -- 172.21.15.3:0/665644363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dfc194e50 con 0x7f4dfc0ff160 2023-07-20T01:36:49.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.284+0000 7f4deeffd700 1 -- 172.21.15.3:0/665644363 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f4df0003d30 con 0x7f4dfc0ff160 2023-07-20T01:36:49.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.284+0000 7f4e03eaa700 1 -- 172.21.15.3:0/665644363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dfc18dc70 con 0x7f4dfc0ff160 2023-07-20T01:36:49.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.284+0000 7f4deeffd700 1 --2- 172.21.15.3:0/665644363 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4de80387b0 0x7f4de803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:49.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.284+0000 7f4deeffd700 1 -- 172.21.15.3:0/665644363 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4df004c4f0 con 0x7f4dfc0ff160 2023-07-20T01:36:49.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.284+0000 7f4e01445700 1 --2- 172.21.15.3:0/665644363 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4de80387b0 0x7f4de803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:49.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.285+0000 7f4e01445700 1 --2- 172.21.15.3:0/665644363 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4de80387b0 0x7f4de803ac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4df8006fd0 tx=0x7f4df8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:49.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.289+0000 7f4deeffd700 1 -- 172.21.15.3:0/665644363 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4df0029360 con 0x7f4dfc0ff160 2023-07-20T01:36:49.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.489+0000 7f4e03eaa700 1 -- 172.21.15.3:0/665644363 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dfc060c40 con 0x7f4de80387b0 2023-07-20T01:36:49.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.493+0000 7f4deeffd700 1 -- 172.21.15.3:0/665644363 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4dfc060c40 con 0x7f4de80387b0 2023-07-20T01:36:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.495+0000 7f4e03eaa700 1 -- 172.21.15.3:0/665644363 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4de80387b0 msgr2=0x7f4de803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.495+0000 7f4e03eaa700 1 --2- 172.21.15.3:0/665644363 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4de80387b0 0x7f4de803ac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4df8006fd0 tx=0x7f4df8006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.495+0000 7f4e03eaa700 1 -- 172.21.15.3:0/665644363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dfc0ff160 msgr2=0x7f4dfc194270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.495+0000 7f4e03eaa700 1 --2- 172.21.15.3:0/665644363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dfc0ff160 0x7f4dfc194270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4df0004f40 tx=0x7f4df00057c0 comp rx=0 tx=0).stop 2023-07-20T01:36:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.495+0000 7f4e03eaa700 1 -- 172.21.15.3:0/665644363 shutdown_connections 2023-07-20T01:36:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.495+0000 7f4e03eaa700 1 --2- 172.21.15.3:0/665644363 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4de80387b0 0x7f4de803ac70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:49.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.495+0000 7f4e03eaa700 1 --2- 172.21.15.3:0/665644363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dfc0ff160 0x7f4dfc194270 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:49.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.495+0000 7f4e03eaa700 1 -- 172.21.15.3:0/665644363 >> 172.21.15.3:0/665644363 conn(0x7f4dfc0fa6e0 msgr2=0x7f4dfc0fb480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:49.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.495+0000 7f4e03eaa700 1 -- 172.21.15.3:0/665644363 shutdown_connections 2023-07-20T01:36:49.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:49.495+0000 7f4e03eaa700 1 -- 172.21.15.3:0/665644363 wait complete. 2023-07-20T01:36:49.987 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi043 2023-07-20T01:36:49.988 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2023-07-20T01:36:49.988 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/etc/ceph/ceph.conf 2023-07-20T01:36:50.009 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2023-07-20T01:36:50.010 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:36:50.070 INFO:tasks.cephadm:Adding host smithi043 to orchestrator... 2023-07-20T01:36:50.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch host add smithi043 2023-07-20T01:36:50.644 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:36:50.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:50 smithi003 ceph-mon[96422]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:36:50.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:50 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:50.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:50 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:36:50.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:50 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:36:50.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:50 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:36:50.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:50 smithi003 ceph-mon[96422]: Updating smithi003:/etc/ceph/ceph.conf 2023-07-20T01:36:50.823 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:50 smithi003 ceph-mon[96422]: Updating smithi003:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:36:51.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.869+0000 7f741b7cf700 1 -- 172.21.15.3:0/3755533575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7414105180 msgr2=0x7f7414105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:51.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.869+0000 7f741b7cf700 1 --2- 172.21.15.3:0/3755533575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7414105180 0x7f7414105560 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7410009910 tx=0x7f7410009c20 comp rx=0 tx=0).stop 2023-07-20T01:36:51.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.869+0000 7f741b7cf700 1 -- 172.21.15.3:0/3755533575 shutdown_connections 2023-07-20T01:36:51.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.869+0000 7f741b7cf700 1 --2- 172.21.15.3:0/3755533575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7414105180 0x7f7414105560 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:51.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.869+0000 7f741b7cf700 1 -- 172.21.15.3:0/3755533575 >> 172.21.15.3:0/3755533575 conn(0x7f7414074c70 msgr2=0x7f7414075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:51.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.870+0000 7f741b7cf700 1 -- 172.21.15.3:0/3755533575 shutdown_connections 2023-07-20T01:36:51.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.870+0000 7f741b7cf700 1 -- 172.21.15.3:0/3755533575 wait complete. 2023-07-20T01:36:51.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.870+0000 7f741b7cf700 1 Processor -- start 2023-07-20T01:36:51.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.870+0000 7f741b7cf700 1 -- start start 2023-07-20T01:36:51.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.870+0000 7f741b7cf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7414105180 0x7f7414196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:51.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.870+0000 7f741b7cf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7414068940 con 0x7f7414105180 2023-07-20T01:36:51.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.870+0000 7f741956b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7414105180 0x7f7414196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:51.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.870+0000 7f741956b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7414105180 0x7f7414196500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41798/0 (socket says 172.21.15.3:41798) 2023-07-20T01:36:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.870+0000 7f741956b700 1 -- 172.21.15.3:0/1327274078 learned_addr learned my addr 172.21.15.3:0/1327274078 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.871+0000 7f741956b700 1 -- 172.21.15.3:0/1327274078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74100095f0 con 0x7f7414105180 2023-07-20T01:36:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.871+0000 7f741956b700 1 --2- 172.21.15.3:0/1327274078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7414105180 0x7f7414196500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7410006010 tx=0x7f7410004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.871+0000 7f740a7fc700 1 -- 172.21.15.3:0/1327274078 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f741001d070 con 0x7f7414105180 2023-07-20T01:36:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.871+0000 7f740a7fc700 1 -- 172.21.15.3:0/1327274078 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f741000bc10 con 0x7f7414105180 2023-07-20T01:36:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.871+0000 7f741b7cf700 1 -- 172.21.15.3:0/1327274078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7414196aa0 con 0x7f7414105180 2023-07-20T01:36:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.871+0000 7f740a7fc700 1 -- 172.21.15.3:0/1327274078 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7410003910 con 0x7f7414105180 2023-07-20T01:36:51.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.871+0000 7f741b7cf700 1 -- 172.21.15.3:0/1327274078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7414196f20 con 0x7f7414105180 2023-07-20T01:36:51.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.872+0000 7f740a7fc700 1 -- 172.21.15.3:0/1327274078 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f7410003bd0 con 0x7f7414105180 2023-07-20T01:36:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.872+0000 7f741b7cf700 1 -- 172.21.15.3:0/1327274078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74141044d0 con 0x7f7414105180 2023-07-20T01:36:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.872+0000 7f740a7fc700 1 --2- 172.21.15.3:0/1327274078 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f74000387b0 0x7f740003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.872+0000 7f740a7fc700 1 -- 172.21.15.3:0/1327274078 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f741004c550 con 0x7f7414105180 2023-07-20T01:36:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.872+0000 7f7418d6a700 1 --2- 172.21.15.3:0/1327274078 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f74000387b0 0x7f740003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.873+0000 7f7418d6a700 1 --2- 172.21.15.3:0/1327274078 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f74000387b0 0x7f740003ac70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7404006fd0 tx=0x7f7404006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:51.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:51.877+0000 7f740a7fc700 1 -- 172.21.15.3:0/1327274078 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f741000ba70 con 0x7f7414105180 2023-07-20T01:36:51.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:51 smithi003 ceph-mon[96422]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:36:51.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:51 smithi003 ceph-mon[96422]: Updating smithi003:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T01:36:51.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:51 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:51.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:51 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:51.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:51 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:52.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:52.085+0000 7f741b7cf700 1 -- 172.21.15.3:0/1327274078 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi043", "target": ["mon-mgr", ""]}) v1 -- 0x7f7414197220 con 0x7f74000387b0 2023-07-20T01:36:53.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:53 smithi003 ceph-mon[96422]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi043", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:36:53.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:53 smithi003 ceph-mon[96422]: Deploying cephadm binary to smithi043 2023-07-20T01:36:54.909 INFO:teuthology.orchestra.run.smithi003.stdout:Added host 'smithi043' with addr '172.21.15.43' 2023-07-20T01:36:54.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:54.939+0000 7f740a7fc700 1 -- 172.21.15.3:0/1327274078 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f7414197220 con 0x7f74000387b0 2023-07-20T01:36:54.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:54.941+0000 7f741b7cf700 1 -- 172.21.15.3:0/1327274078 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f74000387b0 msgr2=0x7f740003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:54.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:54.941+0000 7f741b7cf700 1 --2- 172.21.15.3:0/1327274078 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f74000387b0 0x7f740003ac70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7404006fd0 tx=0x7f7404006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:54.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:54.941+0000 7f741b7cf700 1 -- 172.21.15.3:0/1327274078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7414105180 msgr2=0x7f7414196500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:54.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:54.941+0000 7f741b7cf700 1 --2- 172.21.15.3:0/1327274078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7414105180 0x7f7414196500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7410006010 tx=0x7f7410004ff0 comp rx=0 tx=0).stop 2023-07-20T01:36:54.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:54.941+0000 7f741b7cf700 1 -- 172.21.15.3:0/1327274078 shutdown_connections 2023-07-20T01:36:54.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:54.941+0000 7f741b7cf700 1 --2- 172.21.15.3:0/1327274078 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f74000387b0 0x7f740003ac70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:54.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:54.941+0000 7f741b7cf700 1 --2- 172.21.15.3:0/1327274078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7414105180 0x7f7414196500 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:54.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:54.941+0000 7f741b7cf700 1 -- 172.21.15.3:0/1327274078 >> 172.21.15.3:0/1327274078 conn(0x7f7414074c70 msgr2=0x7f74140fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:54.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:54.941+0000 7f741b7cf700 1 -- 172.21.15.3:0/1327274078 shutdown_connections 2023-07-20T01:36:54.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:54.941+0000 7f741b7cf700 1 -- 172.21.15.3:0/1327274078 wait complete. 2023-07-20T01:36:55.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T01:36:55.874 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:36:56.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:55 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:56.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:55 smithi003 ceph-mon[96422]: Added host smithi043 2023-07-20T01:36:56.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:55 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:36:56.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:55 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:56.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:55 smithi003 ceph-mon[96422]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:36:57.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.104+0000 7f5a708c3700 1 -- 172.21.15.3:0/1319619102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a6c105180 msgr2=0x7f5a6c105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:57.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.104+0000 7f5a708c3700 1 --2- 172.21.15.3:0/1319619102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a6c105180 0x7f5a6c105560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009910 tx=0x7f5a5c009c20 comp rx=0 tx=0).stop 2023-07-20T01:36:57.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.104+0000 7f5a708c3700 1 -- 172.21.15.3:0/1319619102 shutdown_connections 2023-07-20T01:36:57.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.104+0000 7f5a708c3700 1 --2- 172.21.15.3:0/1319619102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a6c105180 0x7f5a6c105560 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:57.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.104+0000 7f5a708c3700 1 -- 172.21.15.3:0/1319619102 >> 172.21.15.3:0/1319619102 conn(0x7f5a6c074c70 msgr2=0x7f5a6c075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:57.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.104+0000 7f5a708c3700 1 -- 172.21.15.3:0/1319619102 shutdown_connections 2023-07-20T01:36:57.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.104+0000 7f5a708c3700 1 -- 172.21.15.3:0/1319619102 wait complete. 2023-07-20T01:36:57.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.105+0000 7f5a708c3700 1 Processor -- start 2023-07-20T01:36:57.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.105+0000 7f5a708c3700 1 -- start start 2023-07-20T01:36:57.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.105+0000 7f5a708c3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a6c105180 0x7f5a6c1964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:57.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.105+0000 7f5a708c3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c196a20 con 0x7f5a6c105180 2023-07-20T01:36:57.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.105+0000 7f5a6a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a6c105180 0x7f5a6c1964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:57.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.105+0000 7f5a6a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a6c105180 0x7f5a6c1964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41824/0 (socket says 172.21.15.3:41824) 2023-07-20T01:36:57.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.105+0000 7f5a6a59c700 1 -- 172.21.15.3:0/1972595222 learned_addr learned my addr 172.21.15.3:0/1972595222 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:57.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.106+0000 7f5a6a59c700 1 -- 172.21.15.3:0/1972595222 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c0095f0 con 0x7f5a6c105180 2023-07-20T01:36:57.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.106+0000 7f5a6a59c700 1 --2- 172.21.15.3:0/1972595222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a6c105180 0x7f5a6c1964e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c006010 tx=0x7f5a5c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:57.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.106+0000 7f5a5b7fe700 1 -- 172.21.15.3:0/1972595222 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a5c01d070 con 0x7f5a6c105180 2023-07-20T01:36:57.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.106+0000 7f5a708c3700 1 -- 172.21.15.3:0/1972595222 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c196c80 con 0x7f5a6c105180 2023-07-20T01:36:57.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.106+0000 7f5a5b7fe700 1 -- 172.21.15.3:0/1972595222 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a5c00bbb0 con 0x7f5a6c105180 2023-07-20T01:36:57.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.106+0000 7f5a5b7fe700 1 -- 172.21.15.3:0/1972595222 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a5c003930 con 0x7f5a6c105180 2023-07-20T01:36:57.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.106+0000 7f5a708c3700 1 -- 172.21.15.3:0/1972595222 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c197100 con 0x7f5a6c105180 2023-07-20T01:36:57.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.107+0000 7f5a5b7fe700 1 -- 172.21.15.3:0/1972595222 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f5a5c003b70 con 0x7f5a6c105180 2023-07-20T01:36:57.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.107+0000 7f5a5b7fe700 1 --2- 172.21.15.3:0/1972595222 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a540387b0 0x7f5a5403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:57.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.107+0000 7f5a5b7fe700 1 -- 172.21.15.3:0/1972595222 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5a5c04c510 con 0x7f5a6c105180 2023-07-20T01:36:57.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.107+0000 7f5a69d9b700 1 --2- 172.21.15.3:0/1972595222 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a540387b0 0x7f5a5403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:57.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.107+0000 7f5a708c3700 1 -- 172.21.15.3:0/1972595222 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c197340 con 0x7f5a6c105180 2023-07-20T01:36:57.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.108+0000 7f5a69d9b700 1 --2- 172.21.15.3:0/1972595222 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a540387b0 0x7f5a5403ac70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5a60006fd0 tx=0x7f5a60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:57.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.112+0000 7f5a5b7fe700 1 -- 172.21.15.3:0/1972595222 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5a5c026020 con 0x7f5a6c105180 2023-07-20T01:36:57.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:57 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:57.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.315+0000 7f5a708c3700 1 -- 172.21.15.3:0/1972595222 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a6c067f80 con 0x7f5a540387b0 2023-07-20T01:36:57.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.316+0000 7f5a5b7fe700 1 -- 172.21.15.3:0/1972595222 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+158 (secure 0 0 0) 0x7f5a6c067f80 con 0x7f5a540387b0 2023-07-20T01:36:57.286 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:36:57.286 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "172.21.15.43", "hostname": "smithi043", "labels": [], "status": ""}] 2023-07-20T01:36:57.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.318+0000 7f5a708c3700 1 -- 172.21.15.3:0/1972595222 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a540387b0 msgr2=0x7f5a5403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:57.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.318+0000 7f5a708c3700 1 --2- 172.21.15.3:0/1972595222 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a540387b0 0x7f5a5403ac70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5a60006fd0 tx=0x7f5a60006e40 comp rx=0 tx=0).stop 2023-07-20T01:36:57.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.318+0000 7f5a708c3700 1 -- 172.21.15.3:0/1972595222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a6c105180 msgr2=0x7f5a6c1964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:57.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.318+0000 7f5a708c3700 1 --2- 172.21.15.3:0/1972595222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a6c105180 0x7f5a6c1964e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c006010 tx=0x7f5a5c004ff0 comp rx=0 tx=0).stop 2023-07-20T01:36:57.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.318+0000 7f5a708c3700 1 -- 172.21.15.3:0/1972595222 shutdown_connections 2023-07-20T01:36:57.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.318+0000 7f5a708c3700 1 --2- 172.21.15.3:0/1972595222 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a540387b0 0x7f5a5403ac70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:57.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.318+0000 7f5a708c3700 1 --2- 172.21.15.3:0/1972595222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a6c105180 0x7f5a6c1964e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:57.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.318+0000 7f5a708c3700 1 -- 172.21.15.3:0/1972595222 >> 172.21.15.3:0/1972595222 conn(0x7f5a6c074c70 msgr2=0x7f5a6c0fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:57.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.318+0000 7f5a708c3700 1 -- 172.21.15.3:0/1972595222 shutdown_connections 2023-07-20T01:36:57.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:57.318+0000 7f5a708c3700 1 -- 172.21.15.3:0/1972595222 wait complete. 2023-07-20T01:36:57.720 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi086 2023-07-20T01:36:57.720 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-07-20T01:36:57.721 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.conf 2023-07-20T01:36:57.741 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-07-20T01:36:57.741 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:36:57.801 INFO:tasks.cephadm:Adding host smithi086 to orchestrator... 2023-07-20T01:36:57.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch host add smithi086 2023-07-20T01:36:58.360 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:36:58.554 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:58 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:36:58.554 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:58 smithi003 ceph-mon[96422]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T01:36:58.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:58 smithi003 ceph-mon[96422]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:36:59.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.617+0000 7f0e14440700 1 -- 172.21.15.3:0/1291234737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c104870 msgr2=0x7f0e0c104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:36:59.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.617+0000 7f0e14440700 1 --2- 172.21.15.3:0/1291234737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c104870 0x7f0e0c104c50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009910 tx=0x7f0e08009c20 comp rx=0 tx=0).stop 2023-07-20T01:36:59.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.617+0000 7f0e14440700 1 -- 172.21.15.3:0/1291234737 shutdown_connections 2023-07-20T01:36:59.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.617+0000 7f0e14440700 1 --2- 172.21.15.3:0/1291234737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c104870 0x7f0e0c104c50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:36:59.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.617+0000 7f0e14440700 1 -- 172.21.15.3:0/1291234737 >> 172.21.15.3:0/1291234737 conn(0x7f0e0c0757d0 msgr2=0x7f0e0c075be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:36:59.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.617+0000 7f0e14440700 1 -- 172.21.15.3:0/1291234737 shutdown_connections 2023-07-20T01:36:59.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.617+0000 7f0e14440700 1 -- 172.21.15.3:0/1291234737 wait complete. 2023-07-20T01:36:59.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.618+0000 7f0e14440700 1 Processor -- start 2023-07-20T01:36:59.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.618+0000 7f0e14440700 1 -- start start 2023-07-20T01:36:59.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.618+0000 7f0e14440700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c104870 0x7f0e0c196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:59.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.618+0000 7f0e14440700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1969c0 con 0x7f0e0c104870 2023-07-20T01:36:59.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.618+0000 7f0e121dc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c104870 0x7f0e0c196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:59.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.618+0000 7f0e121dc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c104870 0x7f0e0c196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41830/0 (socket says 172.21.15.3:41830) 2023-07-20T01:36:59.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.618+0000 7f0e121dc700 1 -- 172.21.15.3:0/4228595771 learned_addr learned my addr 172.21.15.3:0/4228595771 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:36:59.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.619+0000 7f0e121dc700 1 -- 172.21.15.3:0/4228595771 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e080095f0 con 0x7f0e0c104870 2023-07-20T01:36:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.619+0000 7f0e121dc700 1 --2- 172.21.15.3:0/4228595771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c104870 0x7f0e0c196480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0e08004f40 tx=0x7f0e08005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.619+0000 7f0e037fe700 1 -- 172.21.15.3:0/4228595771 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e0801d070 con 0x7f0e0c104870 2023-07-20T01:36:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.619+0000 7f0e14440700 1 -- 172.21.15.3:0/4228595771 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0c196c20 con 0x7f0e0c104870 2023-07-20T01:36:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.619+0000 7f0e037fe700 1 -- 172.21.15.3:0/4228595771 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e08004370 con 0x7f0e0c104870 2023-07-20T01:36:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.619+0000 7f0e037fe700 1 -- 172.21.15.3:0/4228595771 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e08003970 con 0x7f0e0c104870 2023-07-20T01:36:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.619+0000 7f0e14440700 1 -- 172.21.15.3:0/4228595771 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c1970a0 con 0x7f0e0c104870 2023-07-20T01:36:59.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.620+0000 7f0e037fe700 1 -- 172.21.15.3:0/4228595771 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f0e08003c30 con 0x7f0e0c104870 2023-07-20T01:36:59.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.620+0000 7f0e037fe700 1 --2- 172.21.15.3:0/4228595771 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0df80387b0 0x7f0df803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:36:59.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.620+0000 7f0e037fe700 1 -- 172.21.15.3:0/4228595771 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0e0804c4f0 con 0x7f0e0c104870 2023-07-20T01:36:59.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.620+0000 7f0e14440700 1 -- 172.21.15.3:0/4228595771 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0c1972e0 con 0x7f0e0c104870 2023-07-20T01:36:59.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.620+0000 7f0e119db700 1 --2- 172.21.15.3:0/4228595771 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0df80387b0 0x7f0df803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:36:59.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.620+0000 7f0e119db700 1 --2- 172.21.15.3:0/4228595771 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0df80387b0 0x7f0df803ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc006fd0 tx=0x7f0dfc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:36:59.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.625+0000 7f0e037fe700 1 -- 172.21.15.3:0/4228595771 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0e08026070 con 0x7f0e0c104870 2023-07-20T01:36:59.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:36:59.827+0000 7f0e14440700 1 -- 172.21.15.3:0/4228595771 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e0c107d70 con 0x7f0df80387b0 2023-07-20T01:37:00.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:36:59 smithi003 ceph-mon[96422]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:01.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:37:01.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:01.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:01.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:01.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:01.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:37:01.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:01.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:01.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: Updating smithi043:/etc/ceph/ceph.conf 2023-07-20T01:37:01.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: Updating smithi043:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:01.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: Deploying cephadm binary to smithi086 2023-07-20T01:37:01.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:01 smithi003 ceph-mon[96422]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:37:02.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:02 smithi003 ceph-mon[96422]: Updating smithi043:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T01:37:02.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:02 smithi003 ceph-mon[96422]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:02.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:02.698+0000 7f0e037fe700 1 -- 172.21.15.3:0/4228595771 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f0e0c107d70 con 0x7f0df80387b0 2023-07-20T01:37:02.670 INFO:teuthology.orchestra.run.smithi003.stdout:Added host 'smithi086' with addr '172.21.15.86' 2023-07-20T01:37:02.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:02.701+0000 7f0e14440700 1 -- 172.21.15.3:0/4228595771 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0df80387b0 msgr2=0x7f0df803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:02.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:02.701+0000 7f0e14440700 1 --2- 172.21.15.3:0/4228595771 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0df80387b0 0x7f0df803ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc006fd0 tx=0x7f0dfc006e40 comp rx=0 tx=0).stop 2023-07-20T01:37:02.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:02.701+0000 7f0e14440700 1 -- 172.21.15.3:0/4228595771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c104870 msgr2=0x7f0e0c196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:02.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:02.701+0000 7f0e14440700 1 --2- 172.21.15.3:0/4228595771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c104870 0x7f0e0c196480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0e08004f40 tx=0x7f0e08005020 comp rx=0 tx=0).stop 2023-07-20T01:37:02.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:02.701+0000 7f0e14440700 1 -- 172.21.15.3:0/4228595771 shutdown_connections 2023-07-20T01:37:02.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:02.701+0000 7f0e14440700 1 --2- 172.21.15.3:0/4228595771 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0df80387b0 0x7f0df803ac70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:02.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:02.701+0000 7f0e14440700 1 --2- 172.21.15.3:0/4228595771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c104870 0x7f0e0c196480 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:02.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:02.701+0000 7f0e14440700 1 -- 172.21.15.3:0/4228595771 >> 172.21.15.3:0/4228595771 conn(0x7f0e0c0757d0 msgr2=0x7f0e0c102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:02.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:02.701+0000 7f0e14440700 1 -- 172.21.15.3:0/4228595771 shutdown_connections 2023-07-20T01:37:02.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:02.701+0000 7f0e14440700 1 -- 172.21.15.3:0/4228595771 wait complete. 2023-07-20T01:37:03.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T01:37:03.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:03.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:03.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:03.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:03.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:03 smithi003 ceph-mon[96422]: Added host smithi086 2023-07-20T01:37:03.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:03.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:03.667 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:37:04.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.866+0000 7f54205a4700 1 -- 172.21.15.3:0/2241349028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5418105180 msgr2=0x7f5418105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:04.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.866+0000 7f54205a4700 1 --2- 172.21.15.3:0/2241349028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5418105180 0x7f5418105560 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5408009910 tx=0x7f5408009c20 comp rx=0 tx=0).stop 2023-07-20T01:37:04.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.866+0000 7f54205a4700 1 -- 172.21.15.3:0/2241349028 shutdown_connections 2023-07-20T01:37:04.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.866+0000 7f54205a4700 1 --2- 172.21.15.3:0/2241349028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5418105180 0x7f5418105560 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:04.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.866+0000 7f54205a4700 1 -- 172.21.15.3:0/2241349028 >> 172.21.15.3:0/2241349028 conn(0x7f5418074c70 msgr2=0x7f5418075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:04.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.866+0000 7f54205a4700 1 -- 172.21.15.3:0/2241349028 shutdown_connections 2023-07-20T01:37:04.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.866+0000 7f54205a4700 1 -- 172.21.15.3:0/2241349028 wait complete. 2023-07-20T01:37:04.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.867+0000 7f54205a4700 1 Processor -- start 2023-07-20T01:37:04.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.867+0000 7f54205a4700 1 -- start start 2023-07-20T01:37:04.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.867+0000 7f54205a4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5418105180 0x7f54181964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:04.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.867+0000 7f54205a4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54181969e0 con 0x7f5418105180 2023-07-20T01:37:04.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.867+0000 7f541e340700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5418105180 0x7f54181964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:04.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.867+0000 7f541e340700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5418105180 0x7f54181964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41856/0 (socket says 172.21.15.3:41856) 2023-07-20T01:37:04.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.867+0000 7f541e340700 1 -- 172.21.15.3:0/2417441683 learned_addr learned my addr 172.21.15.3:0/2417441683 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:37:04.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.868+0000 7f541e340700 1 -- 172.21.15.3:0/2417441683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54080095f0 con 0x7f5418105180 2023-07-20T01:37:04.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.868+0000 7f541e340700 1 --2- 172.21.15.3:0/2417441683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5418105180 0x7f54181964a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5408004f40 tx=0x7f5408005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:04.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.868+0000 7f540f7fe700 1 -- 172.21.15.3:0/2417441683 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f540801d070 con 0x7f5418105180 2023-07-20T01:37:04.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.868+0000 7f540f7fe700 1 -- 172.21.15.3:0/2417441683 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5408004370 con 0x7f5418105180 2023-07-20T01:37:04.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.868+0000 7f54205a4700 1 -- 172.21.15.3:0/2417441683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5418196c40 con 0x7f5418105180 2023-07-20T01:37:04.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.868+0000 7f540f7fe700 1 -- 172.21.15.3:0/2417441683 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5408003970 con 0x7f5418105180 2023-07-20T01:37:04.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.868+0000 7f54205a4700 1 -- 172.21.15.3:0/2417441683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54181970c0 con 0x7f5418105180 2023-07-20T01:37:04.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.869+0000 7f54205a4700 1 -- 172.21.15.3:0/2417441683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54181044d0 con 0x7f5418105180 2023-07-20T01:37:04.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.869+0000 7f540f7fe700 1 -- 172.21.15.3:0/2417441683 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f5408003c30 con 0x7f5418105180 2023-07-20T01:37:04.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.869+0000 7f540f7fe700 1 --2- 172.21.15.3:0/2417441683 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54040387b0 0x7f540403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:04.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.869+0000 7f540f7fe700 1 -- 172.21.15.3:0/2417441683 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f540804c400 con 0x7f5418105180 2023-07-20T01:37:04.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.869+0000 7f541db3f700 1 --2- 172.21.15.3:0/2417441683 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54040387b0 0x7f540403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:04.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.870+0000 7f541db3f700 1 --2- 172.21.15.3:0/2417441683 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54040387b0 0x7f540403ac70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5414006fd0 tx=0x7f5414006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:04.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:04.874+0000 7f540f7fe700 1 -- 172.21.15.3:0/2417441683 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f540800e9b0 con 0x7f5418105180 2023-07-20T01:37:04.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:04 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:04.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:04 smithi003 ceph-mon[96422]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:05.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.074+0000 7f54205a4700 1 -- 172.21.15.3:0/2417441683 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5418068090 con 0x7f54040387b0 2023-07-20T01:37:05.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.076+0000 7f540f7fe700 1 -- 172.21.15.3:0/2417441683 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+237 (secure 0 0 0) 0x7f5418068090 con 0x7f54040387b0 2023-07-20T01:37:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:37:05.047 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "172.21.15.43", "hostname": "smithi043", "labels": [], "status": ""}, {"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}] 2023-07-20T01:37:05.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.078+0000 7f54205a4700 1 -- 172.21.15.3:0/2417441683 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54040387b0 msgr2=0x7f540403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:05.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.078+0000 7f54205a4700 1 --2- 172.21.15.3:0/2417441683 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54040387b0 0x7f540403ac70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5414006fd0 tx=0x7f5414006e40 comp rx=0 tx=0).stop 2023-07-20T01:37:05.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.078+0000 7f54205a4700 1 -- 172.21.15.3:0/2417441683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5418105180 msgr2=0x7f54181964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:05.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.078+0000 7f54205a4700 1 --2- 172.21.15.3:0/2417441683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5418105180 0x7f54181964a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5408004f40 tx=0x7f5408005020 comp rx=0 tx=0).stop 2023-07-20T01:37:05.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.078+0000 7f54205a4700 1 -- 172.21.15.3:0/2417441683 shutdown_connections 2023-07-20T01:37:05.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.078+0000 7f54205a4700 1 --2- 172.21.15.3:0/2417441683 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54040387b0 0x7f540403ac70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:05.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.078+0000 7f54205a4700 1 --2- 172.21.15.3:0/2417441683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5418105180 0x7f54181964a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:05.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.078+0000 7f54205a4700 1 -- 172.21.15.3:0/2417441683 >> 172.21.15.3:0/2417441683 conn(0x7f5418074c70 msgr2=0x7f54180fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:05.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.078+0000 7f54205a4700 1 -- 172.21.15.3:0/2417441683 shutdown_connections 2023-07-20T01:37:05.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:05.078+0000 7f54205a4700 1 -- 172.21.15.3:0/2417441683 wait complete. 2023-07-20T01:37:05.514 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-20T01:37:05.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph osd crush tunables default 2023-07-20T01:37:06.073 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:37:06.323 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:06 smithi003 ceph-mon[96422]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T01:37:06.324 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:06 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:06.324 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:06 smithi003 ceph-mon[96422]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:07.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.427+0000 7f32d2b68700 1 -- 172.21.15.3:0/1048048654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32cc100450 msgr2=0x7f32cc100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:07.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.427+0000 7f32d2b68700 1 --2- 172.21.15.3:0/1048048654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32cc100450 0x7f32cc100830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009910 tx=0x7f32c0009c20 comp rx=0 tx=0).stop 2023-07-20T01:37:07.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.428+0000 7f32d2b68700 1 -- 172.21.15.3:0/1048048654 shutdown_connections 2023-07-20T01:37:07.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.428+0000 7f32d2b68700 1 --2- 172.21.15.3:0/1048048654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32cc100450 0x7f32cc100830 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:07.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.428+0000 7f32d2b68700 1 -- 172.21.15.3:0/1048048654 >> 172.21.15.3:0/1048048654 conn(0x7f32cc0fbd00 msgr2=0x7f32cc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:07.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.428+0000 7f32d2b68700 1 -- 172.21.15.3:0/1048048654 shutdown_connections 2023-07-20T01:37:07.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.428+0000 7f32d2b68700 1 -- 172.21.15.3:0/1048048654 wait complete. 2023-07-20T01:37:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.428+0000 7f32d2b68700 1 Processor -- start 2023-07-20T01:37:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.428+0000 7f32d2b68700 1 -- start start 2023-07-20T01:37:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.428+0000 7f32d2b68700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32cc100450 0x7f32cc1956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.428+0000 7f32d2b68700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc195c00 con 0x7f32cc100450 2023-07-20T01:37:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.428+0000 7f32d0904700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32cc100450 0x7f32cc1956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.429+0000 7f32d0904700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32cc100450 0x7f32cc1956c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41862/0 (socket says 172.21.15.3:41862) 2023-07-20T01:37:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.429+0000 7f32d0904700 1 -- 172.21.15.3:0/2846892619 learned_addr learned my addr 172.21.15.3:0/2846892619 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:37:07.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.429+0000 7f32d0904700 1 -- 172.21.15.3:0/2846892619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c00095f0 con 0x7f32cc100450 2023-07-20T01:37:07.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.429+0000 7f32d0904700 1 --2- 172.21.15.3:0/2846892619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32cc100450 0x7f32cc1956c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f32c0004f40 tx=0x7f32c0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:07.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.429+0000 7f32c9ffb700 1 -- 172.21.15.3:0/2846892619 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32c001d070 con 0x7f32cc100450 2023-07-20T01:37:07.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.429+0000 7f32c9ffb700 1 -- 172.21.15.3:0/2846892619 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32c0004370 con 0x7f32cc100450 2023-07-20T01:37:07.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.429+0000 7f32d2b68700 1 -- 172.21.15.3:0/2846892619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc195e00 con 0x7f32cc100450 2023-07-20T01:37:07.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.429+0000 7f32c9ffb700 1 -- 172.21.15.3:0/2846892619 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32c00039b0 con 0x7f32cc100450 2023-07-20T01:37:07.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.429+0000 7f32d2b68700 1 -- 172.21.15.3:0/2846892619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32cc196220 con 0x7f32cc100450 2023-07-20T01:37:07.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.430+0000 7f32c9ffb700 1 -- 172.21.15.3:0/2846892619 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f32c0003c70 con 0x7f32cc100450 2023-07-20T01:37:07.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.430+0000 7f32c9ffb700 1 --2- 172.21.15.3:0/2846892619 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f32bc0387b0 0x7f32bc03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:07.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.430+0000 7f32c9ffb700 1 -- 172.21.15.3:0/2846892619 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f32c004c550 con 0x7f32cc100450 2023-07-20T01:37:07.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.430+0000 7f32d2b68700 1 -- 172.21.15.3:0/2846892619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32cc061da0 con 0x7f32cc100450 2023-07-20T01:37:07.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.430+0000 7f32cbfff700 1 --2- 172.21.15.3:0/2846892619 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f32bc0387b0 0x7f32bc03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:07.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.431+0000 7f32cbfff700 1 --2- 172.21.15.3:0/2846892619 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f32bc0387b0 0x7f32bc03ac70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f32b8006fd0 tx=0x7f32b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:07.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.435+0000 7f32c9ffb700 1 -- 172.21.15.3:0/2846892619 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f32c0026070 con 0x7f32cc100450 2023-07-20T01:37:07.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:07.638+0000 7f32d2b68700 1 -- 172.21.15.3:0/2846892619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f32cc108140 con 0x7f32cc100450 2023-07-20T01:37:08.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:08.514+0000 7f32c9ffb700 1 -- 172.21.15.3:0/2846892619 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f32c0049090 con 0x7f32cc100450 2023-07-20T01:37:08.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:08.517+0000 7f32d2b68700 1 -- 172.21.15.3:0/2846892619 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f32bc0387b0 msgr2=0x7f32bc03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:08.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:08.517+0000 7f32d2b68700 1 --2- 172.21.15.3:0/2846892619 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f32bc0387b0 0x7f32bc03ac70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f32b8006fd0 tx=0x7f32b8006e40 comp rx=0 tx=0).stop 2023-07-20T01:37:08.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:08.517+0000 7f32d2b68700 1 -- 172.21.15.3:0/2846892619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32cc100450 msgr2=0x7f32cc1956c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:08.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:08.517+0000 7f32d2b68700 1 --2- 172.21.15.3:0/2846892619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32cc100450 0x7f32cc1956c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f32c0004f40 tx=0x7f32c0005020 comp rx=0 tx=0).stop 2023-07-20T01:37:08.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:08.517+0000 7f32d2b68700 1 -- 172.21.15.3:0/2846892619 shutdown_connections 2023-07-20T01:37:08.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:08.517+0000 7f32d2b68700 1 --2- 172.21.15.3:0/2846892619 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f32bc0387b0 0x7f32bc03ac70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:08.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:08.517+0000 7f32d2b68700 1 --2- 172.21.15.3:0/2846892619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32cc100450 0x7f32cc1956c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:08.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:08.517+0000 7f32d2b68700 1 -- 172.21.15.3:0/2846892619 >> 172.21.15.3:0/2846892619 conn(0x7f32cc0fbd00 msgr2=0x7f32cc1046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:08.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:08.517+0000 7f32d2b68700 1 -- 172.21.15.3:0/2846892619 shutdown_connections 2023-07-20T01:37:08.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:08.517+0000 7f32d2b68700 1 -- 172.21.15.3:0/2846892619 wait complete. 2023-07-20T01:37:08.489 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2023-07-20T01:37:08.824 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:08.824 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:08.824 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:08.825 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:08.825 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:37:08.825 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:08.825 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:08.825 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: Updating smithi086:/etc/ceph/ceph.conf 2023-07-20T01:37:08.825 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2846892619' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T01:37:08.825 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: Updating smithi086:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:08.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:08.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:08.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:08.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:08.827 INFO:tasks.cephadm:Adding mon.a on smithi003 2023-07-20T01:37:08.827 INFO:tasks.cephadm:Adding mon.b on smithi043 2023-07-20T01:37:08.827 INFO:tasks.cephadm:Adding mon.c on smithi086 2023-07-20T01:37:08.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch apply mon '3;smithi003:172.21.15.3=a;smithi043:172.21.15.43=b;smithi086:172.21.15.86=c' 2023-07-20T01:37:09.395 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:09.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:09 smithi003 ceph-mon[96422]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:37:09.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:09 smithi003 ceph-mon[96422]: Updating smithi086:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T01:37:09.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:09 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2846892619' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T01:37:09.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:09 smithi003 ceph-mon[96422]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:10.452 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.481+0000 7f9b413c7700 1 -- 172.21.15.86:0/3347433142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b3c1052a0 msgr2=0x7f9b3c105680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:10.452 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.481+0000 7f9b413c7700 1 --2- 172.21.15.86:0/3347433142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b3c1052a0 0x7f9b3c105680 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009910 tx=0x7f9b30009c20 comp rx=0 tx=0).stop 2023-07-20T01:37:10.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.481+0000 7f9b413c7700 1 -- 172.21.15.86:0/3347433142 shutdown_connections 2023-07-20T01:37:10.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.481+0000 7f9b413c7700 1 --2- 172.21.15.86:0/3347433142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b3c1052a0 0x7f9b3c105680 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:10.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.481+0000 7f9b413c7700 1 -- 172.21.15.86:0/3347433142 >> 172.21.15.86:0/3347433142 conn(0x7f9b3c075860 msgr2=0x7f9b3c075c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:10.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.481+0000 7f9b413c7700 1 -- 172.21.15.86:0/3347433142 shutdown_connections 2023-07-20T01:37:10.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.481+0000 7f9b413c7700 1 -- 172.21.15.86:0/3347433142 wait complete. 2023-07-20T01:37:10.454 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.482+0000 7f9b413c7700 1 Processor -- start 2023-07-20T01:37:10.454 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.482+0000 7f9b413c7700 1 -- start start 2023-07-20T01:37:10.455 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.482+0000 7f9b413c7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b3c1052a0 0x7f9b3c1964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:10.455 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.482+0000 7f9b413c7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c196a00 con 0x7f9b3c1052a0 2023-07-20T01:37:10.455 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.483+0000 7f9b3affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b3c1052a0 0x7f9b3c1964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:10.455 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.483+0000 7f9b3affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b3c1052a0 0x7f9b3c1964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.86:44036/0 (socket says 172.21.15.86:44036) 2023-07-20T01:37:10.455 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.483+0000 7f9b3affd700 1 -- 172.21.15.86:0/1904766146 learned_addr learned my addr 172.21.15.86:0/1904766146 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-07-20T01:37:10.455 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.483+0000 7f9b3affd700 1 -- 172.21.15.86:0/1904766146 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b300095f0 con 0x7f9b3c1052a0 2023-07-20T01:37:10.455 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.483+0000 7f9b3affd700 1 --2- 172.21.15.86:0/1904766146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b3c1052a0 0x7f9b3c1964c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9b30004f40 tx=0x7f9b30005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:10.456 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.483+0000 7f9b23fff700 1 -- 172.21.15.86:0/1904766146 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b3001d070 con 0x7f9b3c1052a0 2023-07-20T01:37:10.456 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.483+0000 7f9b413c7700 1 -- 172.21.15.86:0/1904766146 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3c196c00 con 0x7f9b3c1052a0 2023-07-20T01:37:10.456 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.483+0000 7f9b23fff700 1 -- 172.21.15.86:0/1904766146 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b30004370 con 0x7f9b3c1052a0 2023-07-20T01:37:10.456 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.483+0000 7f9b23fff700 1 -- 172.21.15.86:0/1904766146 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b300039b0 con 0x7f9b3c1052a0 2023-07-20T01:37:10.456 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.483+0000 7f9b413c7700 1 -- 172.21.15.86:0/1904766146 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3c197080 con 0x7f9b3c1052a0 2023-07-20T01:37:10.457 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.484+0000 7f9b413c7700 1 -- 172.21.15.86:0/1904766146 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3c103500 con 0x7f9b3c1052a0 2023-07-20T01:37:10.457 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.484+0000 7f9b23fff700 1 -- 172.21.15.86:0/1904766146 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f9b30003c70 con 0x7f9b3c1052a0 2023-07-20T01:37:10.457 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.485+0000 7f9b23fff700 1 --2- 172.21.15.86:0/1904766146 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f9b240387b0 0x7f9b2403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:10.457 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.485+0000 7f9b23fff700 1 -- 172.21.15.86:0/1904766146 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9b3004c4f0 con 0x7f9b3c1052a0 2023-07-20T01:37:10.458 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.485+0000 7f9b3a7fc700 1 --2- 172.21.15.86:0/1904766146 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f9b240387b0 0x7f9b2403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:10.458 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.485+0000 7f9b3a7fc700 1 --2- 172.21.15.86:0/1904766146 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f9b240387b0 0x7f9b2403ac70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c006fd0 tx=0x7f9b2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:10.460 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.490+0000 7f9b23fff700 1 -- 172.21.15.86:0/1904766146 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9b3002a590 con 0x7f9b3c1052a0 2023-07-20T01:37:10.652 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.681+0000 7f9b413c7700 1 -- 172.21.15.86:0/1904766146 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi043:172.21.15.43=b;smithi086:172.21.15.86=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b3c0fefd0 con 0x7f9b240387b0 2023-07-20T01:37:10.662 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.691+0000 7f9b23fff700 1 -- 172.21.15.86:0/1904766146 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9b3c0fefd0 con 0x7f9b240387b0 2023-07-20T01:37:10.662 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mon update... 2023-07-20T01:37:10.663 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.693+0000 7f9b413c7700 1 -- 172.21.15.86:0/1904766146 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f9b240387b0 msgr2=0x7f9b2403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:10.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.693+0000 7f9b413c7700 1 --2- 172.21.15.86:0/1904766146 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f9b240387b0 0x7f9b2403ac70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c006fd0 tx=0x7f9b2c006e40 comp rx=0 tx=0).stop 2023-07-20T01:37:10.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.693+0000 7f9b413c7700 1 -- 172.21.15.86:0/1904766146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b3c1052a0 msgr2=0x7f9b3c1964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:10.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.693+0000 7f9b413c7700 1 --2- 172.21.15.86:0/1904766146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b3c1052a0 0x7f9b3c1964c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9b30004f40 tx=0x7f9b30005020 comp rx=0 tx=0).stop 2023-07-20T01:37:10.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.693+0000 7f9b413c7700 1 -- 172.21.15.86:0/1904766146 shutdown_connections 2023-07-20T01:37:10.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.693+0000 7f9b413c7700 1 --2- 172.21.15.86:0/1904766146 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f9b240387b0 0x7f9b2403ac70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:10.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.693+0000 7f9b413c7700 1 --2- 172.21.15.86:0/1904766146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b3c1052a0 0x7f9b3c1964c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:10.664 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.693+0000 7f9b413c7700 1 -- 172.21.15.86:0/1904766146 >> 172.21.15.86:0/1904766146 conn(0x7f9b3c075860 msgr2=0x7f9b3c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:10.665 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.693+0000 7f9b413c7700 1 -- 172.21.15.86:0/1904766146 shutdown_connections 2023-07-20T01:37:10.665 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:10.693+0000 7f9b413c7700 1 -- 172.21.15.86:0/1904766146 wait complete. 2023-07-20T01:37:10.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:10 smithi003 ceph-mon[96422]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:10.972 DEBUG:teuthology.orchestra.run.smithi043:mon.b> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.b.service 2023-07-20T01:37:10.974 DEBUG:teuthology.orchestra.run.smithi086:mon.c> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.c.service 2023-07-20T01:37:10.979 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T01:37:10.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T01:37:11.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:-- Logs begin at Thu 2023-07-20 01:20:49 UTC. -- 2023-07-20T01:37:11.011 INFO:journalctl@ceph.mon.b.smithi043.stdout:-- Logs begin at Thu 2023-07-20 01:20:44 UTC. -- 2023-07-20T01:37:11.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:11 smithi003 ceph-mon[96422]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi043:172.21.15.43=b;smithi086:172.21.15.86=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:37:11.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:11 smithi003 ceph-mon[96422]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi043:172.21.15.43=b;smithi086:172.21.15.86=c;count:3 2023-07-20T01:37:11.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:11.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:11.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:11.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:11.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:11.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:11.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:11.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:11 smithi003 ceph-mon[96422]: Deploying daemon mon.c on smithi086 2023-07-20T01:37:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:12.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:12 smithi003 ceph-mon[96422]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:13.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.591+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3023828441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6c81052a0 msgr2=0x7fa6c8105680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:13.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.591+0000 7fa6ce17f700 1 --2- 172.21.15.86:0/3023828441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6c81052a0 0x7fa6c8105680 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8009910 tx=0x7fa6b8009c20 comp rx=0 tx=0).stop 2023-07-20T01:37:13.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.591+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3023828441 shutdown_connections 2023-07-20T01:37:13.564 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.591+0000 7fa6ce17f700 1 --2- 172.21.15.86:0/3023828441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6c81052a0 0x7fa6c8105680 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:13.564 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.591+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3023828441 >> 172.21.15.86:0/3023828441 conn(0x7fa6c8075860 msgr2=0x7fa6c8075c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:13.564 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.591+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3023828441 shutdown_connections 2023-07-20T01:37:13.564 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.591+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3023828441 wait complete. 2023-07-20T01:37:13.564 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.592+0000 7fa6ce17f700 1 Processor -- start 2023-07-20T01:37:13.564 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.592+0000 7fa6ce17f700 1 -- start start 2023-07-20T01:37:13.565 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.592+0000 7fa6ce17f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6c81052a0 0x7fa6c8194570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:13.565 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.592+0000 7fa6ce17f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c8194ab0 con 0x7fa6c81052a0 2023-07-20T01:37:13.566 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.593+0000 7fa6c77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6c81052a0 0x7fa6c8194570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:13.566 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.593+0000 7fa6c77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6c81052a0 0x7fa6c8194570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.86:44042/0 (socket says 172.21.15.86:44042) 2023-07-20T01:37:13.567 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.593+0000 7fa6c77fe700 1 -- 172.21.15.86:0/3137849868 learned_addr learned my addr 172.21.15.86:0/3137849868 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-07-20T01:37:13.567 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.593+0000 7fa6c77fe700 1 -- 172.21.15.86:0/3137849868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b80095f0 con 0x7fa6c81052a0 2023-07-20T01:37:13.567 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.593+0000 7fa6c77fe700 1 --2- 172.21.15.86:0/3137849868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6c81052a0 0x7fa6c8194570 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8006010 tx=0x7fa6b8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:13.567 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.593+0000 7fa6c4ff9700 1 -- 172.21.15.86:0/3137849868 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6b801d070 con 0x7fa6c81052a0 2023-07-20T01:37:13.568 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.594+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3137849868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c8194d10 con 0x7fa6c81052a0 2023-07-20T01:37:13.568 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.594+0000 7fa6c4ff9700 1 -- 172.21.15.86:0/3137849868 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6b800bc10 con 0x7fa6c81052a0 2023-07-20T01:37:13.568 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.594+0000 7fa6c4ff9700 1 -- 172.21.15.86:0/3137849868 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6b8003910 con 0x7fa6c81052a0 2023-07-20T01:37:13.568 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.594+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3137849868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c8195190 con 0x7fa6c81052a0 2023-07-20T01:37:13.568 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.594+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3137849868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c8061da0 con 0x7fa6c81052a0 2023-07-20T01:37:13.569 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.595+0000 7fa6c4ff9700 1 -- 172.21.15.86:0/3137849868 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fa6b8003bd0 con 0x7fa6c81052a0 2023-07-20T01:37:13.569 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.595+0000 7fa6c4ff9700 1 --2- 172.21.15.86:0/3137849868 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fa6b00387b0 0x7fa6b003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:13.569 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.595+0000 7fa6c4ff9700 1 -- 172.21.15.86:0/3137849868 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa6b804c690 con 0x7fa6c81052a0 2023-07-20T01:37:13.569 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.595+0000 7fa6c6ffd700 1 --2- 172.21.15.86:0/3137849868 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fa6b00387b0 0x7fa6b003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:13.569 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.596+0000 7fa6c6ffd700 1 --2- 172.21.15.86:0/3137849868 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fa6b00387b0 0x7fa6b003ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc006fd0 tx=0x7fa6bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:13.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.601+0000 7fa6c4ff9700 1 -- 172.21.15.86:0/3137849868 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa6b800fd00 con 0x7fa6c81052a0 2023-07-20T01:37:13.696 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 systemd[1]: Starting Ceph mon.c for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:37:13.697 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 podman[98076]: 2023-07-20T01:37:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.872+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3137849868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6c8102fe0 con 0x7fa6c81052a0 2023-07-20T01:37:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.873+0000 7fa6c4ff9700 1 -- 172.21.15.86:0/3137849868 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+732 (secure 0 0 0) 0x7fa6b8026030 con 0x7fa6c81052a0 2023-07-20T01:37:13.844 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:37:13.844 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"9b4fd2fc-269d-11ee-9b34-001a4aab830c","modified":"2023-07-20T01:35:35.250748Z","created":"2023-07-20T01:35:35.250748Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-20T01:37:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.875+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3137849868 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fa6b00387b0 msgr2=0x7fa6b003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.875+0000 7fa6ce17f700 1 --2- 172.21.15.86:0/3137849868 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fa6b00387b0 0x7fa6b003ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc006fd0 tx=0x7fa6bc006e40 comp rx=0 tx=0).stop 2023-07-20T01:37:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.875+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3137849868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6c81052a0 msgr2=0x7fa6c8194570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.875+0000 7fa6ce17f700 1 --2- 172.21.15.86:0/3137849868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6c81052a0 0x7fa6c8194570 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8006010 tx=0x7fa6b8004ff0 comp rx=0 tx=0).stop 2023-07-20T01:37:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.875+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3137849868 shutdown_connections 2023-07-20T01:37:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.875+0000 7fa6ce17f700 1 --2- 172.21.15.86:0/3137849868 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fa6b00387b0 0x7fa6b003ac70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.875+0000 7fa6ce17f700 1 --2- 172.21.15.86:0/3137849868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6c81052a0 0x7fa6c8194570 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.875+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3137849868 >> 172.21.15.86:0/3137849868 conn(0x7fa6c8075860 msgr2=0x7fa6c80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.875+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3137849868 shutdown_connections 2023-07-20T01:37:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:13.875+0000 7fa6ce17f700 1 -- 172.21.15.86:0/3137849868 wait complete. 2023-07-20T01:37:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2023-07-20T01:37:13.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T01:37:13.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T01:37:13.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: pidfile_write: ignore empty --pid-file 2023-07-20T01:37:13.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: load: jerasure load: lrc 2023-07-20T01:37:13.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: RocksDB version: 7.9.2 2023-07-20T01:37:13.993 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Git sha 0 2023-07-20T01:37:13.993 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T01:37:13.993 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: DB SUMMARY 2023-07-20T01:37:13.993 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: DB Session ID: 0BLUM8QBJUGNVM08IMCD 2023-07-20T01:37:13.993 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: CURRENT file: CURRENT 2023-07-20T01:37:13.993 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T01:37:13.993 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T01:37:13.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-20T01:37:13.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-20T01:37:13.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.error_if_exists: 0 2023-07-20T01:37:13.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.create_if_missing: 0 2023-07-20T01:37:13.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.paranoid_checks: 1 2023-07-20T01:37:13.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T01:37:13.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T01:37:13.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T01:37:13.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.env: 0x55d229c84c60 2023-07-20T01:37:13.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T01:37:13.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.info_log: 0x55d22aec32c0 2023-07-20T01:37:13.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T01:37:13.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.statistics: (nil) 2023-07-20T01:37:13.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.use_fsync: 0 2023-07-20T01:37:13.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_log_file_size: 0 2023-07-20T01:37:13.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T01:37:13.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T01:37:13.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T01:37:13.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T01:37:13.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.allow_fallocate: 1 2023-07-20T01:37:13.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T01:37:13.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T01:37:13.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.use_direct_reads: 0 2023-07-20T01:37:13.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T01:37:13.997 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T01:37:13.997 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.db_log_dir: 2023-07-20T01:37:13.997 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.wal_dir: 2023-07-20T01:37:13.997 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T01:37:13.998 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T01:37:13.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T01:37:13.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T01:37:13.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T01:37:13.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T01:37:13.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T01:37:13.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T01:37:13.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.write_buffer_manager: 0x55d22ade43c0 2023-07-20T01:37:14.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T01:37:14.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T01:37:14.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T01:37:14.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T01:37:14.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T01:37:14.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T01:37:14.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T01:37:14.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T01:37:14.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.unordered_write: 0 2023-07-20T01:37:14.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T01:37:14.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T01:37:14.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T01:37:14.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T01:37:14.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.row_cache: None 2023-07-20T01:37:14.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.wal_filter: None 2023-07-20T01:37:14.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T01:37:14.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T01:37:14.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.two_write_queues: 0 2023-07-20T01:37:14.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 bash[98076]: 43d501435a05cb6d71756a349dcf8bc2007b97d7b3522e240a07ffcc45956514 2023-07-20T01:37:14.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T01:37:14.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.wal_compression: 0 2023-07-20T01:37:14.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.atomic_flush: 0 2023-07-20T01:37:14.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T01:37:14.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T01:37:14.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T01:37:14.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.log_readahead_size: 0 2023-07-20T01:37:14.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T01:37:14.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T01:37:14.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T01:37:14.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T01:37:14.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T01:37:14.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T01:37:14.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T01:37:14.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_background_jobs: 2 2023-07-20T01:37:14.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_background_compactions: -1 2023-07-20T01:37:14.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_subcompactions: 1 2023-07-20T01:37:14.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T01:37:14.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T01:37:14.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T01:37:14.005 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T01:37:14.005 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T01:37:14.005 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T01:37:14.005 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T01:37:14.005 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T01:37:14.005 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_open_files: -1 2023-07-20T01:37:14.005 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T01:37:14.005 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T01:37:14.006 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T01:37:14.006 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T01:37:14.006 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_background_flushes: -1 2023-07-20T01:37:14.006 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Compression algorithms supported: 2023-07-20T01:37:14.006 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T01:37:14.006 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: kZSTD supported: 0 2023-07-20T01:37:14.006 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: kXpressCompression supported: 0 2023-07-20T01:37:14.006 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T01:37:14.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: kZlibCompression supported: 1 2023-07-20T01:37:14.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: kSnappyCompression supported: 1 2023-07-20T01:37:14.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: kLZ4Compression supported: 1 2023-07-20T01:37:14.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: kBZip2Compression supported: 0 2023-07-20T01:37:14.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T01:37:14.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T01:37:14.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-20T01:37:14.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T01:37:14.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T01:37:14.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.merge_operator: 2023-07-20T01:37:14.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_filter: None 2023-07-20T01:37:14.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T01:37:14.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T01:37:14.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T01:37:14.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T01:37:14.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d22aec33c0) 2023-07-20T01:37:14.009 INFO:journalctl@ceph.mon.c.smithi086.stdout: cache_index_and_filter_blocks: 1 2023-07-20T01:37:14.009 INFO:journalctl@ceph.mon.c.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T01:37:14.009 INFO:journalctl@ceph.mon.c.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T01:37:14.009 INFO:journalctl@ceph.mon.c.smithi086.stdout: pin_top_level_index_and_filter: 1 2023-07-20T01:37:14.009 INFO:journalctl@ceph.mon.c.smithi086.stdout: index_type: 0 2023-07-20T01:37:14.009 INFO:journalctl@ceph.mon.c.smithi086.stdout: data_block_index_type: 0 2023-07-20T01:37:14.009 INFO:journalctl@ceph.mon.c.smithi086.stdout: index_shortening: 1 2023-07-20T01:37:14.010 INFO:journalctl@ceph.mon.c.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T01:37:14.010 INFO:journalctl@ceph.mon.c.smithi086.stdout: checksum: 4 2023-07-20T01:37:14.010 INFO:journalctl@ceph.mon.c.smithi086.stdout: no_block_cache: 0 2023-07-20T01:37:14.010 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_cache: 0x55d22ae83350 2023-07-20T01:37:14.010 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_cache_name: BinnedLRUCache 2023-07-20T01:37:14.010 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_cache_options: 2023-07-20T01:37:14.010 INFO:journalctl@ceph.mon.c.smithi086.stdout: capacity : 536870912 2023-07-20T01:37:14.010 INFO:journalctl@ceph.mon.c.smithi086.stdout: num_shard_bits : 4 2023-07-20T01:37:14.011 INFO:journalctl@ceph.mon.c.smithi086.stdout: strict_capacity_limit : 0 2023-07-20T01:37:14.011 INFO:journalctl@ceph.mon.c.smithi086.stdout: high_pri_pool_ratio: 0.000 2023-07-20T01:37:14.011 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_cache_compressed: (nil) 2023-07-20T01:37:14.011 INFO:journalctl@ceph.mon.c.smithi086.stdout: persistent_cache: (nil) 2023-07-20T01:37:14.011 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_size: 4096 2023-07-20T01:37:14.011 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_size_deviation: 10 2023-07-20T01:37:14.011 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_restart_interval: 16 2023-07-20T01:37:14.011 INFO:journalctl@ceph.mon.c.smithi086.stdout: index_block_restart_interval: 1 2023-07-20T01:37:14.012 INFO:journalctl@ceph.mon.c.smithi086.stdout: metadata_block_size: 4096 2023-07-20T01:37:14.012 INFO:journalctl@ceph.mon.c.smithi086.stdout: partition_filters: 0 2023-07-20T01:37:14.012 INFO:journalctl@ceph.mon.c.smithi086.stdout: use_delta_encoding: 1 2023-07-20T01:37:14.012 INFO:journalctl@ceph.mon.c.smithi086.stdout: filter_policy: bloomfilter 2023-07-20T01:37:14.012 INFO:journalctl@ceph.mon.c.smithi086.stdout: whole_key_filtering: 1 2023-07-20T01:37:14.012 INFO:journalctl@ceph.mon.c.smithi086.stdout: verify_compression: 0 2023-07-20T01:37:14.012 INFO:journalctl@ceph.mon.c.smithi086.stdout: read_amp_bytes_per_bit: 0 2023-07-20T01:37:14.012 INFO:journalctl@ceph.mon.c.smithi086.stdout: format_version: 5 2023-07-20T01:37:14.013 INFO:journalctl@ceph.mon.c.smithi086.stdout: enable_index_compression: 1 2023-07-20T01:37:14.013 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_align: 0 2023-07-20T01:37:14.013 INFO:journalctl@ceph.mon.c.smithi086.stdout: max_auto_readahead_size: 262144 2023-07-20T01:37:14.013 INFO:journalctl@ceph.mon.c.smithi086.stdout: prepopulate_block_cache: 0 2023-07-20T01:37:14.013 INFO:journalctl@ceph.mon.c.smithi086.stdout: initial_auto_readahead_size: 8192 2023-07-20T01:37:14.013 INFO:journalctl@ceph.mon.c.smithi086.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T01:37:14.013 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T01:37:14.013 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T01:37:14.014 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compression: NoCompression 2023-07-20T01:37:14.014 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T01:37:14.014 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T01:37:14.014 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T01:37:14.014 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.num_levels: 7 2023-07-20T01:37:14.014 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T01:37:14.014 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T01:37:14.014 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T01:37:14.015 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T01:37:14.015 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T01:37:14.015 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T01:37:14.015 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T01:37:14.015 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T01:37:14.015 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T01:37:14.015 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T01:37:14.015 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T01:37:14.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T01:37:14.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T01:37:14.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T01:37:14.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T01:37:14.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T01:37:14.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T01:37:14.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T01:37:14.017 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T01:37:14.017 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T01:37:14.017 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T01:37:14.017 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T01:37:14.017 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T01:37:14.017 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T01:37:14.017 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T01:37:14.017 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T01:37:14.018 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T01:37:14.018 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T01:37:14.018 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T01:37:14.018 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T01:37:14.018 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T01:37:14.018 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T01:37:14.018 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T01:37:14.018 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T01:37:14.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T01:37:14.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T01:37:14.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T01:37:14.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T01:37:14.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T01:37:14.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T01:37:14.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T01:37:14.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T01:37:14.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T01:37:14.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T01:37:14.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T01:37:14.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T01:37:14.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T01:37:14.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T01:37:14.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T01:37:14.021 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T01:37:14.021 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T01:37:14.021 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T01:37:14.021 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T01:37:14.021 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.table_properties_collectors: 2023-07-20T01:37:14.021 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.inplace_update_support: 0 2023-07-20T01:37:14.021 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T01:37:14.021 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T01:37:14.022 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T01:37:14.022 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T01:37:14.022 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.bloom_locality: 0 2023-07-20T01:37:14.022 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.max_successive_merges: 0 2023-07-20T01:37:14.022 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T01:37:14.022 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T01:37:14.022 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T01:37:14.022 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T01:37:14.023 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.ttl: 2592000 2023-07-20T01:37:14.023 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T01:37:14.023 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T01:37:14.023 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T01:37:14.023 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.enable_blob_files: false 2023-07-20T01:37:14.023 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.min_blob_size: 0 2023-07-20T01:37:14.023 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T01:37:14.023 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T01:37:14.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T01:37:14.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T01:37:14.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T01:37:14.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T01:37:14.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T01:37:14.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T01:37:14.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T01:37:14.025 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T01:37:14.025 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6045d5dd-7ee1-4f3a-8ed3-12a5938803e3 2023-07-20T01:37:14.025 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689817033935450, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T01:37:14.025 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T01:37:14.025 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689817033936033, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689817033, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6045d5dd-7ee1-4f3a-8ed3-12a5938803e3", "db_session_id": "0BLUM8QBJUGNVM08IMCD", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T01:37:14.025 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689817033936126, "job": 1, "event": "recovery_finished"} 2023-07-20T01:37:14.025 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T01:37:14.025 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T01:37:14.026 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d22af1c000 2023-07-20T01:37:14.026 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: DB pointer 0x55d22af08000 2023-07-20T01:37:14.026 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T01:37:14.026 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T01:37:14.026 INFO:journalctl@ceph.mon.c.smithi086.stdout: ** DB Stats ** 2023-07-20T01:37:14.026 INFO:journalctl@ceph.mon.c.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T01:37:14.026 INFO:journalctl@ceph.mon.c.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T01:37:14.026 INFO:journalctl@ceph.mon.c.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T01:37:14.027 INFO:journalctl@ceph.mon.c.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T01:37:14.027 INFO:journalctl@ceph.mon.c.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T01:37:14.027 INFO:journalctl@ceph.mon.c.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T01:37:14.027 INFO:journalctl@ceph.mon.c.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T01:37:14.027 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2023-07-20T01:37:14.027 INFO:journalctl@ceph.mon.c.smithi086.stdout: ** Compaction Stats [default] ** 2023-07-20T01:37:14.027 INFO:journalctl@ceph.mon.c.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T01:37:14.028 INFO:journalctl@ceph.mon.c.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T01:37:14.028 INFO:journalctl@ceph.mon.c.smithi086.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:37:14.028 INFO:journalctl@ceph.mon.c.smithi086.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:37:14.028 INFO:journalctl@ceph.mon.c.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:37:14.028 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2023-07-20T01:37:14.028 INFO:journalctl@ceph.mon.c.smithi086.stdout: ** Compaction Stats [default] ** 2023-07-20T01:37:14.028 INFO:journalctl@ceph.mon.c.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T01:37:14.028 INFO:journalctl@ceph.mon.c.smithi086.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T01:37:14.029 INFO:journalctl@ceph.mon.c.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:37:14.029 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2023-07-20T01:37:14.029 INFO:journalctl@ceph.mon.c.smithi086.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T01:37:14.029 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2023-07-20T01:37:14.029 INFO:journalctl@ceph.mon.c.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T01:37:14.029 INFO:journalctl@ceph.mon.c.smithi086.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T01:37:14.029 INFO:journalctl@ceph.mon.c.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T01:37:14.029 INFO:journalctl@ceph.mon.c.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T01:37:14.030 INFO:journalctl@ceph.mon.c.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T01:37:14.030 INFO:journalctl@ceph.mon.c.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T01:37:14.030 INFO:journalctl@ceph.mon.c.smithi086.stdout: Cumulative compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T01:37:14.030 INFO:journalctl@ceph.mon.c.smithi086.stdout: Interval compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T01:37:14.030 INFO:journalctl@ceph.mon.c.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T01:37:14.030 INFO:journalctl@ceph.mon.c.smithi086.stdout: Block cache BinnedLRUCache@0x55d22ae83350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2023-07-20T01:37:14.030 INFO:journalctl@ceph.mon.c.smithi086.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T01:37:14.030 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2023-07-20T01:37:14.031 INFO:journalctl@ceph.mon.c.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T01:37:14.031 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-20T01:37:14.031 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: using public_addr v2:172.21.15.86:0/0 -> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-07-20T01:37:14.031 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: starting mon.c rank -1 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:37:14.031 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(???) e0 preinit fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:37:14.031 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).mds e1 new map 2023-07-20T01:37:14.031 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-20T01:37:14.032 INFO:journalctl@ceph.mon.c.smithi086.stdout: e1 2023-07-20T01:37:14.032 INFO:journalctl@ceph.mon.c.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T01:37:14.032 INFO:journalctl@ceph.mon.c.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T01:37:14.032 INFO:journalctl@ceph.mon.c.smithi086.stdout: legacy client fscid: -1 2023-07-20T01:37:14.032 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2023-07-20T01:37:14.032 INFO:journalctl@ceph.mon.c.smithi086.stdout: No filesystems configured 2023-07-20T01:37:14.032 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T01:37:14.032 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T01:37:14.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T01:37:14.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T01:37:14.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T01:37:14.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T01:37:14.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T01:37:14.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:37:14.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:37:14.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:37:14.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: Added host smithi086 2023-07-20T01:37:14.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:14.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.035 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.035 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:14.035 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T01:37:14.035 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.035 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:14.035 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.035 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.035 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.036 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.036 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:37:14.036 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:14.036 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:14.036 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: Updating smithi086:/etc/ceph/ceph.conf 2023-07-20T01:37:14.036 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2846892619' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T01:37:14.036 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: Updating smithi086:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:14.036 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:14.037 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.037 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.037 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.037 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:37:14.037 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: Updating smithi086:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T01:37:14.037 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2846892619' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T01:37:14.037 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:14.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:14.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi043:172.21.15.43=b;smithi086:172.21.15.86=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:37:14.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi043:172.21.15.43=b;smithi086:172.21.15.86=c;count:3 2023-07-20T01:37:14.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:14.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:14.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:14.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:14.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:14.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:14.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: Deploying daemon mon.c on smithi086 2023-07-20T01:37:14.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:14.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:14.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T01:37:14.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-20T01:37:14.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta expand map: {default=false} 2023-07-20T01:37:14.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta from 'false' to 'false' 2023-07-20T01:37:14.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta expanded map: {default=false} 2023-07-20T01:37:14.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta expand map: {default=info} 2023-07-20T01:37:14.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta from 'info' to 'info' 2023-07-20T01:37:14.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta expanded map: {default=info} 2023-07-20T01:37:14.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta expand map: {default=daemon} 2023-07-20T01:37:14.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T01:37:14.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T01:37:14.041 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta expand map: {default=debug} 2023-07-20T01:37:14.041 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T01:37:14.041 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:13 smithi086 ceph-mon[98138]: expand_channel_meta expanded map: {default=debug} 2023-07-20T01:37:14.041 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:14 smithi086 systemd[1]: Started Ceph mon.c for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:37:14.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:13 smithi003 ceph-mon[96422]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:15.330 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T01:37:15.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T01:37:15.894 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:37:19.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:37:19.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: mon.a calling monitor election 2023-07-20T01:37:19.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:19.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: mon.c calling monitor election 2023-07-20T01:37:19.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:19.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:37:19.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} 2023-07-20T01:37:19.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: fsmap 2023-07-20T01:37:19.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:19.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: mgrmap e13: x(active, since 43s) 2023-07-20T01:37:19.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: overall HEALTH_OK 2023-07-20T01:37:19.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:19.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:19.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:19.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:19.359 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:19.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:37:19.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: mon.a calling monitor election 2023-07-20T01:37:19.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:19.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: mon.c calling monitor election 2023-07-20T01:37:19.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:19.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:19.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:37:19.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} 2023-07-20T01:37:19.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: fsmap 2023-07-20T01:37:19.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:19.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: mgrmap e13: x(active, since 43s) 2023-07-20T01:37:19.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: overall HEALTH_OK 2023-07-20T01:37:19.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:19.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:19.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:19.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:19.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:19.987 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.017+0000 7f846e53c700 1 -- 172.21.15.86:0/3671768527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84540035c0 msgr2=0x7f8454005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:19.988 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.017+0000 7f846e53c700 1 --2- 172.21.15.86:0/3671768527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84540035c0 0x7f8454005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8468075920 tx=0x7f845800feb0 comp rx=0 tx=0).stop 2023-07-20T01:37:19.988 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.017+0000 7f846e53c700 1 -- 172.21.15.86:0/3671768527 shutdown_connections 2023-07-20T01:37:19.988 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.017+0000 7f846e53c700 1 --2- 172.21.15.86:0/3671768527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84540035c0 0x7f8454005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:19.988 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.017+0000 7f846e53c700 1 -- 172.21.15.86:0/3671768527 >> 172.21.15.86:0/3671768527 conn(0x7f8468074520 msgr2=0x7f8468074930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:19.988 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.017+0000 7f846e53c700 1 -- 172.21.15.86:0/3671768527 shutdown_connections 2023-07-20T01:37:19.988 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.017+0000 7f846e53c700 1 -- 172.21.15.86:0/3671768527 wait complete. 2023-07-20T01:37:19.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.018+0000 7f846e53c700 1 Processor -- start 2023-07-20T01:37:19.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.018+0000 7f846e53c700 1 -- start start 2023-07-20T01:37:19.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.018+0000 7f846e53c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84540035c0 0x7f846819b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:19.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.018+0000 7f846e53c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f846819bb70 0x7f84681956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:19.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.018+0000 7f846e53c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846819c0b0 con 0x7f846819bb70 2023-07-20T01:37:19.990 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.018+0000 7f846e53c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846819c220 con 0x7f84540035c0 2023-07-20T01:37:19.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f8467fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84540035c0 0x7f846819b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:19.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f8467fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84540035c0 0x7f846819b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58370/0 (socket says 172.21.15.86:58370) 2023-07-20T01:37:19.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f8467fff700 1 -- 172.21.15.86:0/3870095596 learned_addr learned my addr 172.21.15.86:0/3870095596 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-07-20T01:37:19.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f84677fe700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f846819bb70 0x7f84681956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:19.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f8467fff700 1 -- 172.21.15.86:0/3870095596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84540035c0 msgr2=0x7f846819b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-20T01:37:19.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f8467fff700 1 -- 172.21.15.86:0/3870095596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84540035c0 msgr2=0x7f846819b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T01:37:19.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f8467fff700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84540035c0 0x7f846819b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T01:37:19.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f8467fff700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84540035c0 0x7f846819b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:37:19.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f84677fe700 1 -- 172.21.15.86:0/3870095596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84540035c0 msgr2=0x7f846819b630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:37:19.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f84677fe700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84540035c0 0x7f846819b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:19.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f84677fe700 1 -- 172.21.15.86:0/3870095596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845800f900 con 0x7f846819bb70 2023-07-20T01:37:19.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f84677fe700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f846819bb70 0x7f84681956b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f845000b500 tx=0x7f845000b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:19.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.019+0000 7f84657fa700 1 -- 172.21.15.86:0/3870095596 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f8450011630 con 0x7f846819bb70 2023-07-20T01:37:19.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.020+0000 7f84657fa700 1 -- 172.21.15.86:0/3870095596 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8450011c50 con 0x7f846819bb70 2023-07-20T01:37:19.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.020+0000 7f846e53c700 1 -- 172.21.15.86:0/3870095596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8468195cb0 con 0x7f846819bb70 2023-07-20T01:37:19.993 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.020+0000 7f84657fa700 1 -- 172.21.15.86:0/3870095596 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f8450010cc0 con 0x7f846819bb70 2023-07-20T01:37:19.993 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.020+0000 7f846e53c700 1 -- 172.21.15.86:0/3870095596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8468196200 con 0x7f846819bb70 2023-07-20T01:37:19.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.021+0000 7f84657fa700 1 -- 172.21.15.86:0/3870095596 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f8450011790 con 0x7f846819bb70 2023-07-20T01:37:19.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.021+0000 7f846e53c700 1 -- 172.21.15.86:0/3870095596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8468196530 con 0x7f846819bb70 2023-07-20T01:37:19.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.021+0000 7f84657fa700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f84480385f0 0x7f844803aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:19.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.021+0000 7f84657fa700 1 -- 172.21.15.86:0/3870095596 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f845004a890 con 0x7f846819bb70 2023-07-20T01:37:19.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.021+0000 7f8467fff700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f84480385f0 0x7f844803aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:19.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.022+0000 7f8467fff700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f84480385f0 0x7f844803aab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f84580059d0 tx=0x7f8458005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:19.997 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.026+0000 7f84657fa700 1 -- 172.21.15.86:0/3870095596 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8450010e20 con 0x7f846819bb70 2023-07-20T01:37:20.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.291+0000 7f846e53c700 1 -- 172.21.15.86:0/3870095596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8468061da0 con 0x7f846819bb70 2023-07-20T01:37:20.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.292+0000 7f84657fa700 1 -- 172.21.15.86:0/3870095596 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+998 (secure 0 0 0) 0x7f8450018780 con 0x7f846819bb70 2023-07-20T01:37:20.262 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:37:20.263 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":2,"fsid":"9b4fd2fc-269d-11ee-9b34-001a4aab830c","modified":"2023-07-20T01:37:13.966574Z","created":"2023-07-20T01:35:35.250748Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T01:37:20.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.294+0000 7f846e53c700 1 -- 172.21.15.86:0/3870095596 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f84480385f0 msgr2=0x7f844803aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:20.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.294+0000 7f846e53c700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f84480385f0 0x7f844803aab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f84580059d0 tx=0x7f8458005960 comp rx=0 tx=0).stop 2023-07-20T01:37:20.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.294+0000 7f846e53c700 1 -- 172.21.15.86:0/3870095596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f846819bb70 msgr2=0x7f84681956b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:20.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.294+0000 7f846e53c700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f846819bb70 0x7f84681956b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f845000b500 tx=0x7f845000b8c0 comp rx=0 tx=0).stop 2023-07-20T01:37:20.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.294+0000 7f846e53c700 1 -- 172.21.15.86:0/3870095596 shutdown_connections 2023-07-20T01:37:20.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.294+0000 7f846e53c700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f84480385f0 0x7f844803aab0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:20.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.294+0000 7f846e53c700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84540035c0 0x7f846819b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:20.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.294+0000 7f846e53c700 1 --2- 172.21.15.86:0/3870095596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f846819bb70 0x7f84681956b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:20.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.294+0000 7f846e53c700 1 -- 172.21.15.86:0/3870095596 >> 172.21.15.86:0/3870095596 conn(0x7f8468074520 msgr2=0x7f84680fd420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:20.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.294+0000 7f846e53c700 1 -- 172.21.15.86:0/3870095596 shutdown_connections 2023-07-20T01:37:20.266 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:20.294+0000 7f846e53c700 1 -- 172.21.15.86:0/3870095596 wait complete. 2023-07-20T01:37:20.266 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 2 2023-07-20T01:37:20.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:20 smithi086 ceph-mon[98138]: Deploying daemon mon.b on smithi043 2023-07-20T01:37:20.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:20 smithi086 ceph-mon[98138]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:20.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:20 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:20.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:20 smithi003 ceph-mon[96422]: Deploying daemon mon.b on smithi043 2023-07-20T01:37:20.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:20 smithi003 ceph-mon[96422]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:20.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:20 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:21.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:21 smithi003 ceph-mon[96422]: from='client.? 172.21.15.86:0/3870095596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:37:21.189 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:37:20 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:37:20.966+0000 7fbdc229f700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-07-20T01:37:21.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:21 smithi086 ceph-mon[98138]: from='client.? 172.21.15.86:0/3870095596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:37:21.657 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T01:37:21.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T01:37:21.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:21 smithi043 systemd[1]: Starting Ceph mon.b for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:37:22.075 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:21 smithi043 podman[97783]: 2023-07-20T01:37:22.076 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T01:37:22.076 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T01:37:22.076 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: pidfile_write: ignore empty --pid-file 2023-07-20T01:37:22.076 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: load: jerasure load: lrc 2023-07-20T01:37:22.076 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: RocksDB version: 7.9.2 2023-07-20T01:37:22.077 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Git sha 0 2023-07-20T01:37:22.077 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T01:37:22.077 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: DB SUMMARY 2023-07-20T01:37:22.077 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: DB Session ID: KWZTYGOA6M5D0I6O250F 2023-07-20T01:37:22.077 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: CURRENT file: CURRENT 2023-07-20T01:37:22.077 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T01:37:22.077 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T01:37:22.078 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-20T01:37:22.078 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-20T01:37:22.078 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.error_if_exists: 0 2023-07-20T01:37:22.078 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.create_if_missing: 0 2023-07-20T01:37:22.078 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.paranoid_checks: 1 2023-07-20T01:37:22.079 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T01:37:22.079 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T01:37:22.079 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T01:37:22.079 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.env: 0x564a1ef1bc60 2023-07-20T01:37:22.079 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T01:37:22.079 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.info_log: 0x564a2058d2c0 2023-07-20T01:37:22.080 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T01:37:22.080 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.statistics: (nil) 2023-07-20T01:37:22.080 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.use_fsync: 0 2023-07-20T01:37:22.080 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_log_file_size: 0 2023-07-20T01:37:22.080 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T01:37:22.080 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T01:37:22.081 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T01:37:22.081 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T01:37:22.081 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.allow_fallocate: 1 2023-07-20T01:37:22.081 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T01:37:22.081 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T01:37:22.081 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.use_direct_reads: 0 2023-07-20T01:37:22.082 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T01:37:22.082 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T01:37:22.082 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.db_log_dir: 2023-07-20T01:37:22.082 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.wal_dir: 2023-07-20T01:37:22.082 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T01:37:22.082 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T01:37:22.083 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T01:37:22.083 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T01:37:22.083 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T01:37:22.083 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T01:37:22.083 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T01:37:22.083 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T01:37:22.083 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.write_buffer_manager: 0x564a204ae3c0 2023-07-20T01:37:22.084 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T01:37:22.084 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T01:37:22.084 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T01:37:22.084 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T01:37:22.084 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T01:37:22.084 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T01:37:22.084 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T01:37:22.084 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T01:37:22.085 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.unordered_write: 0 2023-07-20T01:37:22.085 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T01:37:22.085 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T01:37:22.085 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T01:37:22.085 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T01:37:22.085 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.row_cache: None 2023-07-20T01:37:22.085 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.wal_filter: None 2023-07-20T01:37:22.086 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T01:37:22.086 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 bash[97783]: c5414c4a9b7f44198840730ee56ffc3a195e761a9e4d29e9108f7d159acfde17 2023-07-20T01:37:22.086 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T01:37:22.086 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.two_write_queues: 0 2023-07-20T01:37:22.086 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T01:37:22.086 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.wal_compression: 0 2023-07-20T01:37:22.086 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.atomic_flush: 0 2023-07-20T01:37:22.086 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T01:37:22.087 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T01:37:22.087 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T01:37:22.087 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.log_readahead_size: 0 2023-07-20T01:37:22.087 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T01:37:22.087 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T01:37:22.087 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T01:37:22.087 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T01:37:22.087 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T01:37:22.089 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T01:37:22.089 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T01:37:22.090 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_background_jobs: 2 2023-07-20T01:37:22.090 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_background_compactions: -1 2023-07-20T01:37:22.090 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_subcompactions: 1 2023-07-20T01:37:22.090 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T01:37:22.090 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T01:37:22.090 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T01:37:22.091 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T01:37:22.091 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T01:37:22.091 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T01:37:22.091 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T01:37:22.091 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T01:37:22.091 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_open_files: -1 2023-07-20T01:37:22.091 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T01:37:22.092 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T01:37:22.092 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T01:37:22.092 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T01:37:22.092 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_background_flushes: -1 2023-07-20T01:37:22.092 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Compression algorithms supported: 2023-07-20T01:37:22.092 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T01:37:22.093 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: kZSTD supported: 0 2023-07-20T01:37:22.093 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: kXpressCompression supported: 0 2023-07-20T01:37:22.093 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T01:37:22.093 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: kZlibCompression supported: 1 2023-07-20T01:37:22.093 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: kSnappyCompression supported: 1 2023-07-20T01:37:22.093 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: kLZ4Compression supported: 1 2023-07-20T01:37:22.094 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: kBZip2Compression supported: 0 2023-07-20T01:37:22.094 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T01:37:22.094 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T01:37:22.094 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-20T01:37:22.094 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T01:37:22.094 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T01:37:22.094 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.merge_operator: 2023-07-20T01:37:22.094 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_filter: None 2023-07-20T01:37:22.095 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T01:37:22.095 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T01:37:22.095 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T01:37:22.095 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T01:37:22.095 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564a2058d3c0) 2023-07-20T01:37:22.095 INFO:journalctl@ceph.mon.b.smithi043.stdout: cache_index_and_filter_blocks: 1 2023-07-20T01:37:22.095 INFO:journalctl@ceph.mon.b.smithi043.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T01:37:22.095 INFO:journalctl@ceph.mon.b.smithi043.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T01:37:22.096 INFO:journalctl@ceph.mon.b.smithi043.stdout: pin_top_level_index_and_filter: 1 2023-07-20T01:37:22.096 INFO:journalctl@ceph.mon.b.smithi043.stdout: index_type: 0 2023-07-20T01:37:22.096 INFO:journalctl@ceph.mon.b.smithi043.stdout: data_block_index_type: 0 2023-07-20T01:37:22.096 INFO:journalctl@ceph.mon.b.smithi043.stdout: index_shortening: 1 2023-07-20T01:37:22.096 INFO:journalctl@ceph.mon.b.smithi043.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T01:37:22.096 INFO:journalctl@ceph.mon.b.smithi043.stdout: checksum: 4 2023-07-20T01:37:22.097 INFO:journalctl@ceph.mon.b.smithi043.stdout: no_block_cache: 0 2023-07-20T01:37:22.097 INFO:journalctl@ceph.mon.b.smithi043.stdout: block_cache: 0x564a2054d1f0 2023-07-20T01:37:22.097 INFO:journalctl@ceph.mon.b.smithi043.stdout: block_cache_name: BinnedLRUCache 2023-07-20T01:37:22.097 INFO:journalctl@ceph.mon.b.smithi043.stdout: block_cache_options: 2023-07-20T01:37:22.097 INFO:journalctl@ceph.mon.b.smithi043.stdout: capacity : 536870912 2023-07-20T01:37:22.097 INFO:journalctl@ceph.mon.b.smithi043.stdout: num_shard_bits : 4 2023-07-20T01:37:22.097 INFO:journalctl@ceph.mon.b.smithi043.stdout: strict_capacity_limit : 0 2023-07-20T01:37:22.098 INFO:journalctl@ceph.mon.b.smithi043.stdout: high_pri_pool_ratio: 0.000 2023-07-20T01:37:22.098 INFO:journalctl@ceph.mon.b.smithi043.stdout: block_cache_compressed: (nil) 2023-07-20T01:37:22.098 INFO:journalctl@ceph.mon.b.smithi043.stdout: persistent_cache: (nil) 2023-07-20T01:37:22.098 INFO:journalctl@ceph.mon.b.smithi043.stdout: block_size: 4096 2023-07-20T01:37:22.098 INFO:journalctl@ceph.mon.b.smithi043.stdout: block_size_deviation: 10 2023-07-20T01:37:22.098 INFO:journalctl@ceph.mon.b.smithi043.stdout: block_restart_interval: 16 2023-07-20T01:37:22.098 INFO:journalctl@ceph.mon.b.smithi043.stdout: index_block_restart_interval: 1 2023-07-20T01:37:22.098 INFO:journalctl@ceph.mon.b.smithi043.stdout: metadata_block_size: 4096 2023-07-20T01:37:22.099 INFO:journalctl@ceph.mon.b.smithi043.stdout: partition_filters: 0 2023-07-20T01:37:22.099 INFO:journalctl@ceph.mon.b.smithi043.stdout: use_delta_encoding: 1 2023-07-20T01:37:22.099 INFO:journalctl@ceph.mon.b.smithi043.stdout: filter_policy: bloomfilter 2023-07-20T01:37:22.099 INFO:journalctl@ceph.mon.b.smithi043.stdout: whole_key_filtering: 1 2023-07-20T01:37:22.099 INFO:journalctl@ceph.mon.b.smithi043.stdout: verify_compression: 0 2023-07-20T01:37:22.100 INFO:journalctl@ceph.mon.b.smithi043.stdout: read_amp_bytes_per_bit: 0 2023-07-20T01:37:22.100 INFO:journalctl@ceph.mon.b.smithi043.stdout: format_version: 5 2023-07-20T01:37:22.100 INFO:journalctl@ceph.mon.b.smithi043.stdout: enable_index_compression: 1 2023-07-20T01:37:22.100 INFO:journalctl@ceph.mon.b.smithi043.stdout: block_align: 0 2023-07-20T01:37:22.100 INFO:journalctl@ceph.mon.b.smithi043.stdout: max_auto_readahead_size: 262144 2023-07-20T01:37:22.100 INFO:journalctl@ceph.mon.b.smithi043.stdout: prepopulate_block_cache: 0 2023-07-20T01:37:22.100 INFO:journalctl@ceph.mon.b.smithi043.stdout: initial_auto_readahead_size: 8192 2023-07-20T01:37:22.100 INFO:journalctl@ceph.mon.b.smithi043.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T01:37:22.101 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T01:37:22.101 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T01:37:22.101 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compression: NoCompression 2023-07-20T01:37:22.101 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T01:37:22.101 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T01:37:22.101 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T01:37:22.102 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.num_levels: 7 2023-07-20T01:37:22.102 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T01:37:22.102 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T01:37:22.102 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T01:37:22.102 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T01:37:22.102 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T01:37:22.102 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T01:37:22.102 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T01:37:22.103 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T01:37:22.103 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T01:37:22.103 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T01:37:22.103 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T01:37:22.103 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T01:37:22.104 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T01:37:22.104 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T01:37:22.104 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T01:37:22.104 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T01:37:22.104 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T01:37:22.104 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T01:37:22.104 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T01:37:22.105 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T01:37:22.105 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T01:37:22.105 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T01:37:22.105 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T01:37:22.105 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T01:37:22.105 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T01:37:22.105 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T01:37:22.106 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T01:37:22.106 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T01:37:22.106 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T01:37:22.106 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T01:37:22.106 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T01:37:22.106 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T01:37:22.106 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T01:37:22.106 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T01:37:22.107 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T01:37:22.107 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T01:37:22.107 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T01:37:22.107 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T01:37:22.107 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T01:37:22.107 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T01:37:22.107 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T01:37:22.107 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T01:37:22.108 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T01:37:22.108 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T01:37:22.108 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T01:37:22.108 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T01:37:22.108 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T01:37:22.108 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T01:37:22.108 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T01:37:22.109 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T01:37:22.109 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T01:37:22.109 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T01:37:22.109 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T01:37:22.109 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.table_properties_collectors: 2023-07-20T01:37:22.109 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.inplace_update_support: 0 2023-07-20T01:37:22.110 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T01:37:22.110 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T01:37:22.110 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T01:37:22.110 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T01:37:22.110 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.bloom_locality: 0 2023-07-20T01:37:22.110 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.max_successive_merges: 0 2023-07-20T01:37:22.110 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T01:37:22.110 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T01:37:22.111 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T01:37:22.111 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T01:37:22.111 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.ttl: 2592000 2023-07-20T01:37:22.111 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T01:37:22.111 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T01:37:22.111 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T01:37:22.111 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.enable_blob_files: false 2023-07-20T01:37:22.111 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.min_blob_size: 0 2023-07-20T01:37:22.112 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T01:37:22.112 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T01:37:22.112 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T01:37:22.112 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T01:37:22.112 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T01:37:22.112 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T01:37:22.112 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T01:37:22.113 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T01:37:22.113 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T01:37:22.113 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T01:37:22.113 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2cf44edb-ee67-42d5-9da9-690cca9f4b9c 2023-07-20T01:37:22.113 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689817042029734, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T01:37:22.113 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T01:37:22.113 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689817042030158, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689817042, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2cf44edb-ee67-42d5-9da9-690cca9f4b9c", "db_session_id": "KWZTYGOA6M5D0I6O250F", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T01:37:22.114 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689817042030228, "job": 1, "event": "recovery_finished"} 2023-07-20T01:37:22.114 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T01:37:22.114 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T01:37:22.114 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564a205e6000 2023-07-20T01:37:22.114 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: DB pointer 0x564a205d2000 2023-07-20T01:37:22.114 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T01:37:22.115 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T01:37:22.115 INFO:journalctl@ceph.mon.b.smithi043.stdout: ** DB Stats ** 2023-07-20T01:37:22.115 INFO:journalctl@ceph.mon.b.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T01:37:22.115 INFO:journalctl@ceph.mon.b.smithi043.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T01:37:22.115 INFO:journalctl@ceph.mon.b.smithi043.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T01:37:22.115 INFO:journalctl@ceph.mon.b.smithi043.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T01:37:22.115 INFO:journalctl@ceph.mon.b.smithi043.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T01:37:22.116 INFO:journalctl@ceph.mon.b.smithi043.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T01:37:22.116 INFO:journalctl@ceph.mon.b.smithi043.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T01:37:22.116 INFO:journalctl@ceph.mon.b.smithi043.stdout: 2023-07-20T01:37:22.116 INFO:journalctl@ceph.mon.b.smithi043.stdout: ** Compaction Stats [default] ** 2023-07-20T01:37:22.116 INFO:journalctl@ceph.mon.b.smithi043.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T01:37:22.116 INFO:journalctl@ceph.mon.b.smithi043.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T01:37:22.117 INFO:journalctl@ceph.mon.b.smithi043.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T01:37:22.117 INFO:journalctl@ceph.mon.b.smithi043.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T01:37:22.117 INFO:journalctl@ceph.mon.b.smithi043.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T01:37:22.117 INFO:journalctl@ceph.mon.b.smithi043.stdout: 2023-07-20T01:37:22.117 INFO:journalctl@ceph.mon.b.smithi043.stdout: ** Compaction Stats [default] ** 2023-07-20T01:37:22.117 INFO:journalctl@ceph.mon.b.smithi043.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T01:37:22.117 INFO:journalctl@ceph.mon.b.smithi043.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T01:37:22.118 INFO:journalctl@ceph.mon.b.smithi043.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T01:37:22.118 INFO:journalctl@ceph.mon.b.smithi043.stdout: 2023-07-20T01:37:22.118 INFO:journalctl@ceph.mon.b.smithi043.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T01:37:22.118 INFO:journalctl@ceph.mon.b.smithi043.stdout: 2023-07-20T01:37:22.118 INFO:journalctl@ceph.mon.b.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T01:37:22.118 INFO:journalctl@ceph.mon.b.smithi043.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T01:37:22.118 INFO:journalctl@ceph.mon.b.smithi043.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T01:37:22.119 INFO:journalctl@ceph.mon.b.smithi043.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T01:37:22.119 INFO:journalctl@ceph.mon.b.smithi043.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T01:37:22.119 INFO:journalctl@ceph.mon.b.smithi043.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T01:37:22.119 INFO:journalctl@ceph.mon.b.smithi043.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T01:37:22.119 INFO:journalctl@ceph.mon.b.smithi043.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T01:37:22.120 INFO:journalctl@ceph.mon.b.smithi043.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T01:37:22.120 INFO:journalctl@ceph.mon.b.smithi043.stdout: Block cache BinnedLRUCache@0x564a2054d1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-20T01:37:22.120 INFO:journalctl@ceph.mon.b.smithi043.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T01:37:22.120 INFO:journalctl@ceph.mon.b.smithi043.stdout: 2023-07-20T01:37:22.121 INFO:journalctl@ceph.mon.b.smithi043.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T01:37:22.121 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-20T01:37:22.121 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: using public_addr v2:172.21.15.43:0/0 -> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2023-07-20T01:37:22.121 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: starting mon.b rank -1 at public addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] at bind addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:37:22.121 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(???) e0 preinit fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:37:22.121 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).mds e1 new map 2023-07-20T01:37:22.121 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-20T01:37:22.122 INFO:journalctl@ceph.mon.b.smithi043.stdout: e1 2023-07-20T01:37:22.122 INFO:journalctl@ceph.mon.b.smithi043.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T01:37:22.122 INFO:journalctl@ceph.mon.b.smithi043.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T01:37:22.122 INFO:journalctl@ceph.mon.b.smithi043.stdout: legacy client fscid: -1 2023-07-20T01:37:22.122 INFO:journalctl@ceph.mon.b.smithi043.stdout: 2023-07-20T01:37:22.122 INFO:journalctl@ceph.mon.b.smithi043.stdout: No filesystems configured 2023-07-20T01:37:22.122 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T01:37:22.123 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T01:37:22.123 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T01:37:22.123 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T01:37:22.123 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T01:37:22.123 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T01:37:22.123 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T01:37:22.124 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:37:22.124 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:37:22.124 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:37:22.124 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.124 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.125 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.125 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.125 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: Added host smithi086 2023-07-20T01:37:22.125 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:22.125 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.125 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.125 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:22.126 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T01:37:22.126 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.126 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:22.126 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.126 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.126 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.126 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.126 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:37:22.127 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:22.127 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:22.127 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: Updating smithi086:/etc/ceph/ceph.conf 2023-07-20T01:37:22.127 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2846892619' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T01:37:22.127 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: Updating smithi086:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:22.127 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:22.127 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.128 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.128 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.128 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:37:22.128 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: Updating smithi086:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T01:37:22.128 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2846892619' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T01:37:22.128 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:22.128 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:22.129 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi043:172.21.15.43=b;smithi086:172.21.15.86=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:37:22.129 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi043:172.21.15.43=b;smithi086:172.21.15.86=c;count:3 2023-07-20T01:37:22.129 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.129 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:22.129 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:22.129 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:22.129 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.130 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:22.130 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:22.130 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: Deploying daemon mon.c on smithi086 2023-07-20T01:37:22.130 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:22.130 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:22.130 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:37:22.131 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:22.131 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.a calling monitor election 2023-07-20T01:37:22.131 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:22.131 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:22.131 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:22.131 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.c calling monitor election 2023-07-20T01:37:22.132 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:22.132 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:22.132 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:22.132 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:22.132 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:37:22.132 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} 2023-07-20T01:37:22.132 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: fsmap 2023-07-20T01:37:22.133 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:22.133 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mgrmap e13: x(active, since 43s) 2023-07-20T01:37:22.133 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: overall HEALTH_OK 2023-07-20T01:37:22.133 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.133 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.133 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:22.133 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:22.133 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:22.134 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: Deploying daemon mon.b on smithi043 2023-07-20T01:37:22.134 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:22.134 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:22.134 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: from='client.? 172.21.15.86:0/3870095596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:37:22.134 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T01:37:22.134 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta expand map: {default=false} 2023-07-20T01:37:22.134 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta from 'false' to 'false' 2023-07-20T01:37:22.134 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta expanded map: {default=false} 2023-07-20T01:37:22.135 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta expand map: {default=info} 2023-07-20T01:37:22.135 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta from 'info' to 'info' 2023-07-20T01:37:22.135 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta expanded map: {default=info} 2023-07-20T01:37:22.135 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta expand map: {default=daemon} 2023-07-20T01:37:22.135 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T01:37:22.136 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T01:37:22.136 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta expand map: {default=debug} 2023-07-20T01:37:22.136 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T01:37:22.136 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: expand_channel_meta expanded map: {default=debug} 2023-07-20T01:37:22.136 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 ceph-mon[97823]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-20T01:37:22.136 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:22 smithi043 systemd[1]: Started Ceph mon.b for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:37:22.220 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:37:27.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:37:27.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: mon.a calling monitor election 2023-07-20T01:37:27.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: mon.c calling monitor election 2023-07-20T01:37:27.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:27.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:27.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:27.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:37:27.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} 2023-07-20T01:37:27.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: fsmap 2023-07-20T01:37:27.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:27.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: mgrmap e13: x(active, since 51s) 2023-07-20T01:37:27.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T01:37:27.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T01:37:27.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T01:37:27.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: mon.b (rank 2) addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] is down (out of quorum) 2023-07-20T01:37:27.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:27.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:27.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:37:27.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: mon.a calling monitor election 2023-07-20T01:37:27.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: mon.c calling monitor election 2023-07-20T01:37:27.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:27.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:27.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:27.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:27.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:37:27.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} 2023-07-20T01:37:27.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: fsmap 2023-07-20T01:37:27.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:27.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: mgrmap e13: x(active, since 51s) 2023-07-20T01:37:27.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T01:37:27.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T01:37:27.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T01:37:27.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: mon.b (rank 2) addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] is down (out of quorum) 2023-07-20T01:37:27.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:27.609 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:27.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.707+0000 7f5a998d9700 1 -- 172.21.15.86:0/2717943970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a7c0057d0 msgr2=0x7f5a7c007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:27.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.707+0000 7f5a998d9700 1 --2- 172.21.15.86:0/2717943970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a7c0057d0 0x7f5a7c007c60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5a94075920 tx=0x7f5a8800ce70 comp rx=0 tx=0).stop 2023-07-20T01:37:27.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.707+0000 7f5a998d9700 1 -- 172.21.15.86:0/2717943970 shutdown_connections 2023-07-20T01:37:27.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.707+0000 7f5a998d9700 1 --2- 172.21.15.86:0/2717943970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a7c0057d0 0x7f5a7c007c60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:27.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.707+0000 7f5a998d9700 1 -- 172.21.15.86:0/2717943970 >> 172.21.15.86:0/2717943970 conn(0x7f5a94074520 msgr2=0x7f5a94074930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:27.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.707+0000 7f5a998d9700 1 -- 172.21.15.86:0/2717943970 shutdown_connections 2023-07-20T01:37:27.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.707+0000 7f5a998d9700 1 -- 172.21.15.86:0/2717943970 wait complete. 2023-07-20T01:37:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a998d9700 1 Processor -- start 2023-07-20T01:37:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a998d9700 1 -- start start 2023-07-20T01:37:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a998d9700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a7c0057d0 0x7f5a94199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a998d9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a94199ac0 0x7f5a941936a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a998d9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a94193be0 0x7f5a94194090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a998d9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9419a120 con 0x7f5a94199ac0 2023-07-20T01:37:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a998d9700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a941a03a0 con 0x7f5a7c0057d0 2023-07-20T01:37:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a998d9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a941a0530 con 0x7f5a94193be0 2023-07-20T01:37:27.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a937fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a94193be0 0x7f5a94194090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:27.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a937fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a94193be0 0x7f5a94194090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58400/0 (socket says 172.21.15.86:58400) 2023-07-20T01:37:27.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a937fe700 1 -- 172.21.15.86:0/3396080500 learned_addr learned my addr 172.21.15.86:0/3396080500 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-07-20T01:37:27.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a937fe700 1 -- 172.21.15.86:0/3396080500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a94193be0 msgr2=0x7f5a94194090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-20T01:37:27.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a937fe700 1 -- 172.21.15.86:0/3396080500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a94193be0 msgr2=0x7f5a94194090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T01:37:27.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a937fe700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a94193be0 0x7f5a94194090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T01:37:27.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a937fe700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a94193be0 0x7f5a94194090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:37:27.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.708+0000 7f5a927fc700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a94199ac0 0x7f5a941936a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:27.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a92ffd700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a7c0057d0 0x7f5a94199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:27.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a92ffd700 1 -- 172.21.15.86:0/3396080500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a7c0057d0 msgr2=0x7f5a94199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-20T01:37:27.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a92ffd700 1 -- 172.21.15.86:0/3396080500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a7c0057d0 msgr2=0x7f5a94199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T01:37:27.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a92ffd700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a7c0057d0 0x7f5a94199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T01:37:27.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a92ffd700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a7c0057d0 0x7f5a94199580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:37:27.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a927fc700 1 -- 172.21.15.86:0/3396080500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a94193be0 msgr2=0x7f5a94194090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:37:27.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a927fc700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a94193be0 0x7f5a94194090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:27.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a927fc700 1 -- 172.21.15.86:0/3396080500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a7c0057d0 msgr2=0x7f5a94199580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:37:27.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a927fc700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a7c0057d0 0x7f5a94199580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:27.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a927fc700 1 -- 172.21.15.86:0/3396080500 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8800c8c0 con 0x7f5a94199ac0 2023-07-20T01:37:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a927fc700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a94199ac0 0x7f5a941936a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400b550 tx=0x7f5a8400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a988d7700 1 -- 172.21.15.86:0/3396080500 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84004230 con 0x7f5a94199ac0 2023-07-20T01:37:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.709+0000 7f5a998d9700 1 -- 172.21.15.86:0/3396080500 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a941a0810 con 0x7f5a94199ac0 2023-07-20T01:37:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.710+0000 7f5a998d9700 1 -- 172.21.15.86:0/3396080500 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a941a0d60 con 0x7f5a94199ac0 2023-07-20T01:37:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.710+0000 7f5a988d7700 1 -- 172.21.15.86:0/3396080500 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a84009320 con 0x7f5a94199ac0 2023-07-20T01:37:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.710+0000 7f5a988d7700 1 -- 172.21.15.86:0/3396080500 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84011a30 con 0x7f5a94199ac0 2023-07-20T01:37:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.710+0000 7f5a998d9700 1 -- 172.21.15.86:0/3396080500 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7c002130 con 0x7f5a94199ac0 2023-07-20T01:37:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.710+0000 7f5a988d7700 1 -- 172.21.15.86:0/3396080500 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f5a84011b90 con 0x7f5a94199ac0 2023-07-20T01:37:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.711+0000 7f5a988d7700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a80038c30 0x7f5a8003b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.711+0000 7f5a988d7700 1 -- 172.21.15.86:0/3396080500 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5a8404a9d0 con 0x7f5a94199ac0 2023-07-20T01:37:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.711+0000 7f5a92ffd700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a80038c30 0x7f5a8003b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.711+0000 7f5a92ffd700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a80038c30 0x7f5a8003b0f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5a88001eb0 tx=0x7f5a8800cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:27.716+0000 7f5a988d7700 1 -- 172.21.15.86:0/3396080500 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5a8400d490 con 0x7f5a94199ac0 2023-07-20T01:37:27.975 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.005+0000 7f5a998d9700 1 -- 172.21.15.86:0/3396080500 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a7c0032e0 con 0x7f5a94199ac0 2023-07-20T01:37:27.976 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.006+0000 7f5a988d7700 1 -- 172.21.15.86:0/3396080500 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1262 (secure 0 0 0) 0x7f5a8400f2e0 con 0x7f5a94199ac0 2023-07-20T01:37:27.976 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-07-20T01:37:27.977 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":3,"fsid":"9b4fd2fc-269d-11ee-9b34-001a4aab830c","modified":"2023-07-20T01:37:22.052145Z","created":"2023-07-20T01:35:35.250748Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T01:37:27.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 -- 172.21.15.86:0/3396080500 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a80038c30 msgr2=0x7f5a8003b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:27.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a80038c30 0x7f5a8003b0f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5a88001eb0 tx=0x7f5a8800cf90 comp rx=0 tx=0).stop 2023-07-20T01:37:27.979 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 -- 172.21.15.86:0/3396080500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a94199ac0 msgr2=0x7f5a941936a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:27.979 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a94199ac0 0x7f5a941936a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400b550 tx=0x7f5a8400b910 comp rx=0 tx=0).stop 2023-07-20T01:37:27.979 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 -- 172.21.15.86:0/3396080500 shutdown_connections 2023-07-20T01:37:27.979 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a94193be0 0x7f5a94194090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:27.979 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f5a80038c30 0x7f5a8003b0f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:27.979 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a7c0057d0 0x7f5a94199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:27.979 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 --2- 172.21.15.86:0/3396080500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a94199ac0 0x7f5a941936a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:27.980 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 -- 172.21.15.86:0/3396080500 >> 172.21.15.86:0/3396080500 conn(0x7f5a94074520 msgr2=0x7f5a94100580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:27.980 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 -- 172.21.15.86:0/3396080500 shutdown_connections 2023-07-20T01:37:27.980 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:28.008+0000 7f5a998d9700 1 -- 172.21.15.86:0/3396080500 wait complete. 2023-07-20T01:37:27.980 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 3 2023-07-20T01:37:28.284 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:28.284 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:28.285 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:28.285 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:28 smithi086 ceph-mon[98138]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:28.285 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:28 smithi086 ceph-mon[98138]: from='client.? 172.21.15.86:0/3396080500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:37:28.285 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:28.286 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-20T01:37:28.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph config generate-minimal-conf 2023-07-20T01:37:28.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:28.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:28.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:28.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:28 smithi003 ceph-mon[96422]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:28.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:28 smithi003 ceph-mon[96422]: from='client.? 172.21.15.86:0/3396080500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:37:28.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:37:29.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:37:29.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: mon.a calling monitor election 2023-07-20T01:37:29.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: mon.c calling monitor election 2023-07-20T01:37:29.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:29.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:37:29.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:29.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:29.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:29.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:29.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:29.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:29.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:29.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:37:29.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} 2023-07-20T01:37:29.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: fsmap 2023-07-20T01:37:29.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:29.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: mgrmap e13: x(active, since 51s) 2023-07-20T01:37:29.508 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T01:37:29.508 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T01:37:29.508 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T01:37:29.508 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: mon.b (rank 2) addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] is down (out of quorum) 2023-07-20T01:37:29.508 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:29.508 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:29.509 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:29.509 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:29.509 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:29.509 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:29.509 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='client.? 172.21.15.86:0/3396080500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:37:29.509 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:30.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.069+0000 7ff56c861700 1 -- 172.21.15.3:0/2792675825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff568106290 msgr2=0x7ff568106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:30.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.069+0000 7ff56c861700 1 --2- 172.21.15.3:0/2792675825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff568106290 0x7ff568106670 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff558009910 tx=0x7ff558009c20 comp rx=0 tx=0).stop 2023-07-20T01:37:30.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.069+0000 7ff56c861700 1 -- 172.21.15.3:0/2792675825 shutdown_connections 2023-07-20T01:37:30.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.069+0000 7ff56c861700 1 --2- 172.21.15.3:0/2792675825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff568106290 0x7ff568106670 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:30.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.069+0000 7ff56c861700 1 -- 172.21.15.3:0/2792675825 >> 172.21.15.3:0/2792675825 conn(0x7ff568074c70 msgr2=0x7ff568075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:30.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.069+0000 7ff56c861700 1 -- 172.21.15.3:0/2792675825 shutdown_connections 2023-07-20T01:37:30.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.069+0000 7ff56c861700 1 -- 172.21.15.3:0/2792675825 wait complete. 2023-07-20T01:37:30.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff56c861700 1 Processor -- start 2023-07-20T01:37:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff56c861700 1 -- start start 2023-07-20T01:37:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff56c861700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568106290 0x7ff5680ff1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff56c861700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5680ff720 0x7ff5680ffba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff56c861700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5681a19e0 0x7ff5681a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff56c861700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5681a4420 con 0x7ff5680ff720 2023-07-20T01:37:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff56c861700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5681a4590 con 0x7ff568106290 2023-07-20T01:37:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff56c861700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5681a4750 con 0x7ff5681a19e0 2023-07-20T01:37:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff565d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5680ff720 0x7ff5680ffba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff565d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5680ff720 0x7ff5680ffba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41870/0 (socket says 172.21.15.3:41870) 2023-07-20T01:37:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff565d9b700 1 -- 172.21.15.3:0/4202476723 learned_addr learned my addr 172.21.15.3:0/4202476723 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:37:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff56659c700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568106290 0x7ff5680ff1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.070+0000 7ff566d9d700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5681a19e0 0x7ff5681a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff565d9b700 1 -- 172.21.15.3:0/4202476723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5681a19e0 msgr2=0x7ff5681a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:30.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff565d9b700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5681a19e0 0x7ff5681a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff565d9b700 1 -- 172.21.15.3:0/4202476723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568106290 msgr2=0x7ff5680ff1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff565d9b700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568106290 0x7ff5680ff1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff565d9b700 1 -- 172.21.15.3:0/4202476723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580095f0 con 0x7ff5680ff720 2023-07-20T01:37:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff56659c700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568106290 0x7ff5680ff1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:37:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff565d9b700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5680ff720 0x7ff5680ffba0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff55c00b780 tx=0x7ff55c00bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff5577fe700 1 -- 172.21.15.3:0/4202476723 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c004230 con 0x7ff5680ff720 2023-07-20T01:37:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff5577fe700 1 -- 172.21.15.3:0/4202476723 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff55c004390 con 0x7ff5680ff720 2023-07-20T01:37:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff5577fe700 1 -- 172.21.15.3:0/4202476723 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c011b30 con 0x7ff5680ff720 2023-07-20T01:37:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff56c861700 1 -- 172.21.15.3:0/4202476723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5681a4a30 con 0x7ff5680ff720 2023-07-20T01:37:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.071+0000 7ff56c861700 1 -- 172.21.15.3:0/4202476723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5681a4f30 con 0x7ff5680ff720 2023-07-20T01:37:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.072+0000 7ff5577fe700 1 -- 172.21.15.3:0/4202476723 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7ff55c011cd0 con 0x7ff5680ff720 2023-07-20T01:37:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.072+0000 7ff56c861700 1 -- 172.21.15.3:0/4202476723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff568061da0 con 0x7ff5680ff720 2023-07-20T01:37:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.072+0000 7ff5577fe700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff550038c30 0x7ff55003b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.072+0000 7ff5577fe700 1 -- 172.21.15.3:0/4202476723 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff55c04aeb0 con 0x7ff5680ff720 2023-07-20T01:37:30.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.072+0000 7ff56659c700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff550038c30 0x7ff55003b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:30.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.073+0000 7ff56659c700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff550038c30 0x7ff55003b0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff558006010 tx=0x7ff55800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:30.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.077+0000 7ff5577fe700 1 -- 172.21.15.3:0/4202476723 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff55c01ae40 con 0x7ff5680ff720 2023-07-20T01:37:30.166 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: mon.b calling monitor election 2023-07-20T01:37:30.166 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: Updating smithi003:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:30.166 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: mon.b calling monitor election 2023-07-20T01:37:30.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: mon.c calling monitor election 2023-07-20T01:37:30.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: mon.a calling monitor election 2023-07-20T01:37:30.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: Updating smithi086:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:30.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: Updating smithi043:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:30.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T01:37:30.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} 2023-07-20T01:37:30.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: fsmap 2023-07-20T01:37:30.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:30.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: mgrmap e13: x(active, since 53s) 2023-07-20T01:37:30.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T01:37:30.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: Cluster is now healthy 2023-07-20T01:37:30.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: overall HEALTH_OK 2023-07-20T01:37:30.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T01:37:30.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:30.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:37:30.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:30.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: Reconfiguring daemon mon.a on smithi003 2023-07-20T01:37:30.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:29 smithi003 ceph-mon[96422]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:30.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: mon.b calling monitor election 2023-07-20T01:37:30.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: Updating smithi003:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:30.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: mon.b calling monitor election 2023-07-20T01:37:30.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: mon.c calling monitor election 2023-07-20T01:37:30.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: mon.a calling monitor election 2023-07-20T01:37:30.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: Updating smithi086:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:30.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: Updating smithi043:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:30.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T01:37:30.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} 2023-07-20T01:37:30.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: fsmap 2023-07-20T01:37:30.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:30.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: mgrmap e13: x(active, since 53s) 2023-07-20T01:37:30.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T01:37:30.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: Cluster is now healthy 2023-07-20T01:37:30.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: overall HEALTH_OK 2023-07-20T01:37:30.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.258 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.258 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.258 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.258 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.258 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.258 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T01:37:30.258 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:30.259 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:37:30.259 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:30.259 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: Reconfiguring daemon mon.a on smithi003 2023-07-20T01:37:30.259 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:29 smithi043 ceph-mon[97823]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:30.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.285+0000 7ff56c861700 1 -- 172.21.15.3:0/4202476723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff568100710 con 0x7ff5680ff720 2023-07-20T01:37:30.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.285+0000 7ff5577fe700 1 -- 172.21.15.3:0/4202476723 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+269 (secure 0 0 0) 0x7ff55c00f2e0 con 0x7ff5680ff720 2023-07-20T01:37:30.260 INFO:teuthology.orchestra.run.smithi003.stdout:# minimal ceph.conf for 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:37:30.260 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2023-07-20T01:37:30.261 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:37:30.261 INFO:teuthology.orchestra.run.smithi003.stdout: mon_host = [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2023-07-20T01:37:30.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.287+0000 7ff56c861700 1 -- 172.21.15.3:0/4202476723 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff550038c30 msgr2=0x7ff55003b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:30.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.287+0000 7ff56c861700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff550038c30 0x7ff55003b0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff558006010 tx=0x7ff55800b540 comp rx=0 tx=0).stop 2023-07-20T01:37:30.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.287+0000 7ff56c861700 1 -- 172.21.15.3:0/4202476723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5680ff720 msgr2=0x7ff5680ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:30.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.287+0000 7ff56c861700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5680ff720 0x7ff5680ffba0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff55c00b780 tx=0x7ff55c00bb40 comp rx=0 tx=0).stop 2023-07-20T01:37:30.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.287+0000 7ff56c861700 1 -- 172.21.15.3:0/4202476723 shutdown_connections 2023-07-20T01:37:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.287+0000 7ff56c861700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5681a19e0 0x7ff5681a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.287+0000 7ff56c861700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff550038c30 0x7ff55003b0f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.287+0000 7ff56c861700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568106290 0x7ff5680ff1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.287+0000 7ff56c861700 1 --2- 172.21.15.3:0/4202476723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5680ff720 0x7ff5680ffba0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.287+0000 7ff56c861700 1 -- 172.21.15.3:0/4202476723 >> 172.21.15.3:0/4202476723 conn(0x7ff568074c70 msgr2=0x7ff5680fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.288+0000 7ff56c861700 1 -- 172.21.15.3:0/4202476723 shutdown_connections 2023-07-20T01:37:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:30.288+0000 7ff56c861700 1 -- 172.21.15.3:0/4202476723 wait complete. 2023-07-20T01:37:30.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: mon.b calling monitor election 2023-07-20T01:37:30.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: Updating smithi003:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:30.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: mon.b calling monitor election 2023-07-20T01:37:30.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: mon.c calling monitor election 2023-07-20T01:37:30.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: mon.a calling monitor election 2023-07-20T01:37:30.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: Updating smithi086:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:30.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: Updating smithi043:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:37:30.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T01:37:30.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} 2023-07-20T01:37:30.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: fsmap 2023-07-20T01:37:30.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:37:30.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: mgrmap e13: x(active, since 53s) 2023-07-20T01:37:30.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T01:37:30.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: Cluster is now healthy 2023-07-20T01:37:30.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: overall HEALTH_OK 2023-07-20T01:37:30.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:30.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T01:37:30.359 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:30.359 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:37:30.359 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:30.359 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: Reconfiguring daemon mon.a on smithi003 2023-07-20T01:37:30.359 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:29 smithi086 ceph-mon[98138]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:30.939 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-20T01:37:30.939 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:37:30.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T01:37:30.971 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:37:30.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:37:31.040 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2023-07-20T01:37:31.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T01:37:31.079 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2023-07-20T01:37:31.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:37:31.153 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-07-20T01:37:31.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T01:37:31.185 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:31.185 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:30 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/4202476723' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:31.185 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:31.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:31.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:30 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/4202476723' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:31.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:31.190 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-07-20T01:37:31.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:37:31.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:37:31.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:30 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/4202476723' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:31.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:31.260 INFO:tasks.cephadm:Adding mgr.x on smithi003 2023-07-20T01:37:31.260 INFO:tasks.cephadm:Adding mgr.y on smithi043 2023-07-20T01:37:31.261 INFO:tasks.cephadm:Adding mgr.z on smithi086 2023-07-20T01:37:31.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch apply mgr '3;smithi003=x;smithi043=y;smithi086=z' 2023-07-20T01:37:31.849 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:37:32.337 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:32.337 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:32.337 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:32 smithi043 ceph-mon[97823]: Reconfiguring mon.b (monmap changed)... 2023-07-20T01:37:32.337 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:32.338 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:37:32.338 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:32.338 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:32 smithi043 ceph-mon[97823]: Reconfiguring daemon mon.b on smithi043 2023-07-20T01:37:32.338 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:32 smithi043 ceph-mon[97823]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:32.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:32.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:32.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:32 smithi086 ceph-mon[98138]: Reconfiguring mon.b (monmap changed)... 2023-07-20T01:37:32.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:32.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:37:32.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:32.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:32 smithi086 ceph-mon[98138]: Reconfiguring daemon mon.b on smithi043 2023-07-20T01:37:32.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:32 smithi086 ceph-mon[98138]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:32.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:32.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:32.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:32 smithi003 ceph-mon[96422]: Reconfiguring mon.b (monmap changed)... 2023-07-20T01:37:32.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:32.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:37:32.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:32.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:32 smithi003 ceph-mon[96422]: Reconfiguring daemon mon.b on smithi043 2023-07-20T01:37:32.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:32 smithi003 ceph-mon[96422]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:33.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.337+0000 7f0c5e04a700 1 -- 172.21.15.86:0/3609646269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c580ff270 msgr2=0x7f0c580ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:33.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.337+0000 7f0c5e04a700 1 --2- 172.21.15.86:0/3609646269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c580ff270 0x7f0c580ff690 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009910 tx=0x7f0c48009c20 comp rx=0 tx=0).stop 2023-07-20T01:37:33.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.337+0000 7f0c5e04a700 1 -- 172.21.15.86:0/3609646269 shutdown_connections 2023-07-20T01:37:33.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.337+0000 7f0c5e04a700 1 --2- 172.21.15.86:0/3609646269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c580ff270 0x7f0c580ff690 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:33.310 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.337+0000 7f0c5e04a700 1 -- 172.21.15.86:0/3609646269 >> 172.21.15.86:0/3609646269 conn(0x7f0c580fa810 msgr2=0x7f0c580fcc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:33.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.337+0000 7f0c5e04a700 1 -- 172.21.15.86:0/3609646269 shutdown_connections 2023-07-20T01:37:33.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.337+0000 7f0c5e04a700 1 -- 172.21.15.86:0/3609646269 wait complete. 2023-07-20T01:37:33.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.338+0000 7f0c5e04a700 1 Processor -- start 2023-07-20T01:37:33.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.338+0000 7f0c5e04a700 1 -- start start 2023-07-20T01:37:33.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.338+0000 7f0c5e04a700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c580ff270 0x7f0c5819bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:33.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.338+0000 7f0c5e04a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c5819c170 0x7f0c581a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:33.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.338+0000 7f0c5e04a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c5819c5f0 0x7f0c581a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:33.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.338+0000 7f0c5e04a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c581a3f20 con 0x7f0c5819c170 2023-07-20T01:37:33.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.338+0000 7f0c5e04a700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c581a4090 con 0x7f0c580ff270 2023-07-20T01:37:33.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.338+0000 7f0c5e04a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c581a4250 con 0x7f0c5819c5f0 2023-07-20T01:37:33.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.338+0000 7f0c57fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c5819c5f0 0x7f0c581a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:33.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.338+0000 7f0c57fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c5819c5f0 0x7f0c581a3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58410/0 (socket says 172.21.15.86:58410) 2023-07-20T01:37:33.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.338+0000 7f0c57fff700 1 -- 172.21.15.86:0/1688321664 learned_addr learned my addr 172.21.15.86:0/1688321664 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-07-20T01:37:33.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c56ffd700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c5819c170 0x7f0c581a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:33.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c577fe700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c580ff270 0x7f0c5819bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:33.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c57fff700 1 -- 172.21.15.86:0/1688321664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c580ff270 msgr2=0x7f0c5819bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:33.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c57fff700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c580ff270 0x7f0c5819bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:33.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c57fff700 1 -- 172.21.15.86:0/1688321664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c5819c170 msgr2=0x7f0c581a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:33.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c57fff700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c5819c170 0x7f0c581a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:33.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c57fff700 1 -- 172.21.15.86:0/1688321664 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c480095f0 con 0x7f0c5819c5f0 2023-07-20T01:37:33.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c577fe700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c580ff270 0x7f0c5819bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:37:33.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c57fff700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c5819c5f0 0x7f0c581a3750 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000ba50 tx=0x7f0c5000be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:33.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c54ff9700 1 -- 172.21.15.86:0/1688321664 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000cab0 con 0x7f0c5819c5f0 2023-07-20T01:37:33.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c5e04a700 1 -- 172.21.15.86:0/1688321664 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c581a4530 con 0x7f0c5819c5f0 2023-07-20T01:37:33.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c54ff9700 1 -- 172.21.15.86:0/1688321664 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c5000cc10 con 0x7f0c5819c5f0 2023-07-20T01:37:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c54ff9700 1 -- 172.21.15.86:0/1688321664 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c500075b0 con 0x7f0c5819c5f0 2023-07-20T01:37:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.339+0000 7f0c5e04a700 1 -- 172.21.15.86:0/1688321664 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c581a4a60 con 0x7f0c5819c5f0 2023-07-20T01:37:33.316 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.340+0000 7f0c54ff9700 1 -- 172.21.15.86:0/1688321664 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f0c50007710 con 0x7f0c5819c5f0 2023-07-20T01:37:33.316 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.340+0000 7f0c54ff9700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0c44038c20 0x7f0c4403b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:33.316 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.340+0000 7f0c54ff9700 1 -- 172.21.15.86:0/1688321664 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0c5004ae10 con 0x7f0c5819c5f0 2023-07-20T01:37:33.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.340+0000 7f0c5e04a700 1 -- 172.21.15.86:0/1688321664 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5818cc30 con 0x7f0c5819c5f0 2023-07-20T01:37:33.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.341+0000 7f0c577fe700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0c44038c20 0x7f0c4403b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:33.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.341+0000 7f0c577fe700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0c44038c20 0x7f0c4403b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800b5c0 tx=0x7f0c48005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:33.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.345+0000 7f0c54ff9700 1 -- 172.21.15.86:0/1688321664 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0c50019070 con 0x7f0c5819c5f0 2023-07-20T01:37:33.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.550+0000 7f0c5e04a700 1 -- 172.21.15.86:0/1688321664 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi043=y;smithi086=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c58060c40 con 0x7f0c44038c20 2023-07-20T01:37:33.528 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.558+0000 7f0c54ff9700 1 -- 172.21.15.86:0/1688321664 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0c58060c40 con 0x7f0c44038c20 2023-07-20T01:37:33.528 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mgr update... 2023-07-20T01:37:33.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 -- 172.21.15.86:0/1688321664 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0c44038c20 msgr2=0x7f0c4403b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:33.530 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0c44038c20 0x7f0c4403b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800b5c0 tx=0x7f0c48005fb0 comp rx=0 tx=0).stop 2023-07-20T01:37:33.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 -- 172.21.15.86:0/1688321664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c5819c5f0 msgr2=0x7f0c581a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:33.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c5819c5f0 0x7f0c581a3750 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000ba50 tx=0x7f0c5000be10 comp rx=0 tx=0).stop 2023-07-20T01:37:33.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 -- 172.21.15.86:0/1688321664 shutdown_connections 2023-07-20T01:37:33.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c5819c5f0 0x7f0c581a3750 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:33.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0c44038c20 0x7f0c4403b0e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:33.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c580ff270 0x7f0c5819bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:33.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 --2- 172.21.15.86:0/1688321664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c5819c170 0x7f0c581a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:33.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 -- 172.21.15.86:0/1688321664 >> 172.21.15.86:0/1688321664 conn(0x7f0c580fa810 msgr2=0x7f0c580fb530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:33.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 -- 172.21.15.86:0/1688321664 shutdown_connections 2023-07-20T01:37:33.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:37:33.560+0000 7f0c5e04a700 1 -- 172.21.15.86:0/1688321664 wait complete. 2023-07-20T01:37:33.557 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:33.558 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:33.558 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:33 smithi086 ceph-mon[98138]: Reconfiguring mon.c (monmap changed)... 2023-07-20T01:37:33.558 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:33.558 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:37:33.558 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:33.558 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:33 smithi086 ceph-mon[98138]: Reconfiguring daemon mon.c on smithi086 2023-07-20T01:37:33.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:33.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:33.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:33 smithi003 ceph-mon[96422]: Reconfiguring mon.c (monmap changed)... 2023-07-20T01:37:33.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:33.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:37:33.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:33.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:33 smithi003 ceph-mon[96422]: Reconfiguring daemon mon.c on smithi086 2023-07-20T01:37:33.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:33.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:33.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:33 smithi043 ceph-mon[97823]: Reconfiguring mon.c (monmap changed)... 2023-07-20T01:37:33.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:37:33.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:37:33.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:33.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:33 smithi043 ceph-mon[97823]: Reconfiguring daemon mon.c on smithi086 2023-07-20T01:37:33.871 DEBUG:teuthology.orchestra.run.smithi043:mgr.y> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.y.service 2023-07-20T01:37:33.873 DEBUG:teuthology.orchestra.run.smithi086:mgr.z> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.z.service 2023-07-20T01:37:33.875 INFO:tasks.cephadm:Deploying OSDs... 2023-07-20T01:37:33.876 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-07-20T01:37:33.876 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T01:37:33.893 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T01:37:33.893 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2023-07-20T01:37:33.903 INFO:journalctl@ceph.mgr.z.smithi086.stdout:-- Logs begin at Thu 2023-07-20 01:20:49 UTC. -- 2023-07-20T01:37:33.908 INFO:journalctl@ceph.mgr.y.smithi043.stdout:-- Logs begin at Thu 2023-07-20 01:20:44 UTC. -- 2023-07-20T01:37:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T01:37:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:33.952 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 377087 Links: 1 2023-07-20T01:37:33.952 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:33.952 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:33.952 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-07-20 01:36:46.559137773 +0000 2023-07-20T01:37:33.952 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-07-20 01:30:26.560157863 +0000 2023-07-20T01:37:33.953 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-07-20 01:30:26.560157863 +0000 2023-07-20T01:37:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2023-07-20T01:37:33.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T01:37:34.018 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-07-20T01:37:34.019 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-07-20T01:37:34.019 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000242808 s, 2.1 MB/s 2023-07-20T01:37:34.020 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T01:37:34.079 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2023-07-20T01:37:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T01:37:34.137 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:34.138 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 381078 Links: 1 2023-07-20T01:37:34.138 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:34.138 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:34.138 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-07-20 01:36:46.559137773 +0000 2023-07-20T01:37:34.138 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-07-20 01:30:27.081151411 +0000 2023-07-20T01:37:34.138 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-07-20 01:30:27.081151411 +0000 2023-07-20T01:37:34.138 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2023-07-20T01:37:34.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T01:37:34.205 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-07-20T01:37:34.205 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-07-20T01:37:34.205 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000219715 s, 2.3 MB/s 2023-07-20T01:37:34.206 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T01:37:34.266 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2023-07-20T01:37:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T01:37:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:34.325 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 380041 Links: 1 2023-07-20T01:37:34.325 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:34.325 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:34.325 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-07-20 01:36:46.559137773 +0000 2023-07-20T01:37:34.325 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-07-20 01:30:27.612144836 +0000 2023-07-20T01:37:34.325 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-07-20 01:30:27.612144836 +0000 2023-07-20T01:37:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2023-07-20T01:37:34.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T01:37:34.391 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-07-20T01:37:34.391 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-07-20T01:37:34.392 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.00021768 s, 2.4 MB/s 2023-07-20T01:37:34.393 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T01:37:34.452 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2023-07-20T01:37:34.509 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T01:37:34.510 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:34.510 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 383268 Links: 1 2023-07-20T01:37:34.510 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:34.510 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:34.510 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-07-20 01:36:46.559137773 +0000 2023-07-20T01:37:34.511 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-07-20 01:30:28.152138149 +0000 2023-07-20T01:37:34.511 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-07-20 01:30:28.152138149 +0000 2023-07-20T01:37:34.511 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2023-07-20T01:37:34.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T01:37:34.574 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.575 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.575 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:34.575 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:34.575 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:34.575 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.575 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi043=y;smithi086=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:37:34.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: Saving service mgr spec with placement smithi003=x;smithi043=y;smithi086=z;count:3 2023-07-20T01:37:34.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T01:37:34.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:37:34.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:37:34.577 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:34.577 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: Reconfiguring daemon mgr.x on smithi003 2023-07-20T01:37:34.577 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:34 smithi003 ceph-mon[96422]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-07-20T01:37:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-07-20T01:37:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000234113 s, 2.2 MB/s 2023-07-20T01:37:34.579 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T01:37:34.637 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2023-07-20T01:37:34.637 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T01:37:34.659 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T01:37:34.659 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 2023-07-20T01:37:34.720 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T01:37:34.720 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:34.720 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 366374 Links: 1 2023-07-20T01:37:34.721 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:34.721 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:34.721 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2023-07-20 01:37:21.467265618 +0000 2023-07-20T01:37:34.721 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2023-07-20 01:30:24.948066090 +0000 2023-07-20T01:37:34.721 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2023-07-20 01:30:24.948066090 +0000 2023-07-20T01:37:34.721 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2023-07-20T01:37:34.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T01:37:34.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:34.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:34.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:34.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi043=y;smithi086=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:37:34.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: Saving service mgr spec with placement smithi003=x;smithi043=y;smithi086=z;count:3 2023-07-20T01:37:34.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T01:37:34.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:37:34.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:37:34.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:34.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: Reconfiguring daemon mgr.x on smithi003 2023-07-20T01:37:34.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:34 smithi043 ceph-mon[97823]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2023-07-20T01:37:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2023-07-20T01:37:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000127126 s, 4.0 MB/s 2023-07-20T01:37:34.788 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T01:37:34.849 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 2023-07-20T01:37:34.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:34.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:34.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:34.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi043=y;smithi086=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:37:34.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: Saving service mgr spec with placement smithi003=x;smithi043=y;smithi086=z;count:3 2023-07-20T01:37:34.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:34.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T01:37:34.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:37:34.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:37:34.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:34.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: Reconfiguring daemon mgr.x on smithi003 2023-07-20T01:37:34.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:34 smithi086 ceph-mon[98138]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:34.909 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T01:37:34.910 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:34.910 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 366415 Links: 1 2023-07-20T01:37:34.910 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:34.910 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:34.911 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2023-07-20 01:37:21.467265618 +0000 2023-07-20T01:37:34.911 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2023-07-20 01:30:25.609058335 +0000 2023-07-20T01:37:34.911 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2023-07-20 01:30:25.609058335 +0000 2023-07-20T01:37:34.912 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2023-07-20T01:37:34.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T01:37:34.978 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2023-07-20T01:37:34.979 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2023-07-20T01:37:34.979 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000124303 s, 4.1 MB/s 2023-07-20T01:37:34.980 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T01:37:35.041 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 2023-07-20T01:37:35.100 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T01:37:35.100 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:35.100 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 378180 Links: 1 2023-07-20T01:37:35.101 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:35.101 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:35.101 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2023-07-20 01:37:21.467265618 +0000 2023-07-20T01:37:35.101 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2023-07-20 01:30:26.246050860 +0000 2023-07-20T01:37:35.101 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2023-07-20 01:30:26.246050860 +0000 2023-07-20T01:37:35.101 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2023-07-20T01:37:35.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T01:37:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2023-07-20T01:37:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2023-07-20T01:37:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000124244 s, 4.1 MB/s 2023-07-20T01:37:35.171 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T01:37:35.231 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 2023-07-20T01:37:35.290 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T01:37:35.290 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:35.290 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 383159 Links: 1 2023-07-20T01:37:35.290 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:35.291 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:35.291 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2023-07-20 01:37:21.467265618 +0000 2023-07-20T01:37:35.291 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2023-07-20 01:30:26.784044547 +0000 2023-07-20T01:37:35.291 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2023-07-20 01:30:26.784044547 +0000 2023-07-20T01:37:35.291 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2023-07-20T01:37:35.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T01:37:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2023-07-20T01:37:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2023-07-20T01:37:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000120914 s, 4.2 MB/s 2023-07-20T01:37:35.360 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T01:37:35.420 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-07-20T01:37:35.420 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T01:37:35.438 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T01:37:35.439 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2023-07-20T01:37:35.495 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T01:37:35.495 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:35.495 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 372541 Links: 1 2023-07-20T01:37:35.495 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:35.495 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:35.496 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-07-20 01:37:13.365488617 +0000 2023-07-20T01:37:35.496 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-07-20 01:30:03.714280335 +0000 2023-07-20T01:37:35.496 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-07-20 01:30:03.714280335 +0000 2023-07-20T01:37:35.496 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-07-20T01:37:35.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T01:37:35.562 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-07-20T01:37:35.563 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-07-20T01:37:35.563 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000239733 s, 2.1 MB/s 2023-07-20T01:37:35.564 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T01:37:35.621 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2023-07-20T01:37:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T01:37:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:35.680 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 371479 Links: 1 2023-07-20T01:37:35.680 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:35.680 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:35.680 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-07-20 01:37:13.365488617 +0000 2023-07-20T01:37:35.680 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-07-20 01:30:04.253268688 +0000 2023-07-20T01:37:35.680 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-07-20 01:30:04.253268688 +0000 2023-07-20T01:37:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-07-20T01:37:35.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T01:37:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-07-20T01:37:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-07-20T01:37:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000211723 s, 2.4 MB/s 2023-07-20T01:37:35.747 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T01:37:35.803 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2023-07-20T01:37:35.860 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T01:37:35.861 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:35.861 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 379318 Links: 1 2023-07-20T01:37:35.861 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:35.861 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:35.861 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-07-20 01:37:13.365488617 +0000 2023-07-20T01:37:35.861 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-07-20 01:30:04.772257473 +0000 2023-07-20T01:37:35.862 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-07-20 01:30:04.772257473 +0000 2023-07-20T01:37:35.862 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-07-20T01:37:35.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T01:37:35.928 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-07-20T01:37:35.928 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-07-20T01:37:35.928 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000197938 s, 2.6 MB/s 2023-07-20T01:37:35.929 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T01:37:35.986 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2023-07-20T01:37:36.043 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T01:37:36.044 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:37:36.044 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 375638 Links: 1 2023-07-20T01:37:36.044 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:37:36.044 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:37:36.044 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-07-20 01:37:13.365488617 +0000 2023-07-20T01:37:36.045 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-07-20 01:30:05.300246063 +0000 2023-07-20T01:37:36.045 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-07-20 01:30:05.300246063 +0000 2023-07-20T01:37:36.045 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-07-20T01:37:36.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T01:37:36.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:36.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:36.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:36.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:36.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:36.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:36.107 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:37:36.107 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:37:36.107 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:37:36.107 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:36.108 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: Deploying daemon mgr.z on smithi086 2023-07-20T01:37:36.108 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:35 smithi086 ceph-mon[98138]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:36.109 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-07-20T01:37:36.109 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-07-20T01:37:36.110 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000195768 s, 2.6 MB/s 2023-07-20T01:37:36.111 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T01:37:36.168 INFO:tasks.cephadm:Deploying osd.0 on smithi003 with /dev/vg_nvme/lv_4... 2023-07-20T01:37:36.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T01:37:36.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:36.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:36.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:36.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:36.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:36.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:36.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:37:36.258 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:37:36.258 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:37:36.258 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:36.259 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: Deploying daemon mgr.z on smithi086 2023-07-20T01:37:36.259 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:35 smithi043 ceph-mon[97823]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:36.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:36.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:36.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:36.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:36.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:36.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:36.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:37:36.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:37:36.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:37:36.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:36.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: Deploying daemon mgr.z on smithi086 2023-07-20T01:37:36.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:35 smithi003 ceph-mon[96422]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:36.741 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:37:36.749 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:36 smithi086 systemd[1]: Starting Ceph mgr.z for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:37:37.041 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:36 smithi086 podman[100300]: 2023-07-20T01:37:37.042 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:36 smithi086 bash[100300]: 70e7808bdfff3fed8df93504a8104f81f2c0b736124ea45ce33143f63fe0261e 2023-07-20T01:37:37.042 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:36 smithi086 ceph-mgr[100340]: -- 172.21.15.86:0/2316305158 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56080e4051e0 con 0x56080e3da800 2023-07-20T01:37:37.348 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:37 smithi086 systemd[1]: Started Ceph mgr.z for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:37:37.348 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:37.111+0000 7fe17ac68200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T01:37:37.348 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:37.196+0000 7fe17ac68200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T01:37:37.349 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:37.285+0000 7fe17ac68200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T01:37:37.349 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:37.376+0000 7fe17ac68200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T01:37:37.605 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:37.544+0000 7fe17ac68200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T01:37:38.103 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:37.849+0000 7fe17ac68200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T01:37:38.103 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:37.937+0000 7fe17ac68200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T01:37:38.103 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:38.071+0000 7fe17ac68200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T01:37:38.271 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T01:37:38.271 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T01:37:38.271 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2023-07-20T01:37:38.271 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2023-07-20T01:37:38.271 INFO:teuthology.orchestra.run.smithi003.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108064 s, 970 MB/s 2023-07-20T01:37:38.272 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2023-07-20T01:37:38.332 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:38.333 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:38.333 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:38.333 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:37:38.333 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:37:38.334 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:37:38.334 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:38.334 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:38 smithi043 ceph-mon[97823]: Deploying daemon mgr.y on smithi043 2023-07-20T01:37:38.334 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:38 smithi043 ceph-mon[97823]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:38.355 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:38.166+0000 7fe17ac68200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T01:37:38.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:38.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:38.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:38.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:37:38.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:37:38.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:37:38.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:38.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-mon[98138]: Deploying daemon mgr.y on smithi043 2023-07-20T01:37:38.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-mon[98138]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:38.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:38.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:38.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:38.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:37:38.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:37:38.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:37:38.401 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:38.401 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:38 smithi003 ceph-mon[96422]: Deploying daemon mgr.y on smithi043 2023-07-20T01:37:38.401 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:38 smithi003 ceph-mon[96422]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:38.631 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:38 smithi043 systemd[1]: Starting Ceph mgr.y for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:37:38.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi003:vg_nvme/lv_4 2023-07-20T01:37:38.855 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:38 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:38.580+0000 7fe17ac68200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T01:37:39.218 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:38 smithi043 podman[99290]: 2023-07-20T01:37:39.218 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:39 smithi043 bash[99290]: 5027d1c1b7849b01c3d711cb0c6fa58fb3696b7ed7ab7479205113c35d4a266d 2023-07-20T01:37:39.218 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:39 smithi043 ceph-mgr[99331]: -- 172.21.15.43:0/2321747425 <== mon.2 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558cd59911e0 con 0x558cd5966800 2023-07-20T01:37:39.260 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:37:39.504 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:39 smithi043 systemd[1]: Started Ceph mgr.y for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:37:39.505 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:39.331+0000 7fc059c68200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T01:37:39.505 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:39.427+0000 7fc059c68200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T01:37:39.605 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:39 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:39.147+0000 7fe17ac68200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T01:37:39.605 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:39 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:39.315+0000 7fe17ac68200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T01:37:39.837 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:39.536+0000 7fc059c68200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T01:37:39.837 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:39.653+0000 7fc059c68200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T01:37:40.242 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:39.867+0000 7fc059c68200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T01:37:40.352 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:40.105+0000 7fe17ac68200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:37:40.352 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:40.204+0000 7fe17ac68200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T01:37:40.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:40 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:40 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:40 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:40 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:40 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:40.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:40 smithi086 ceph-mon[98138]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:40.505 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:40.273+0000 7fc059c68200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T01:37:40.505 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:40.386+0000 7fc059c68200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T01:37:40.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:40 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:40 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:40 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:40 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:40 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:40.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:40 smithi043 ceph-mon[97823]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:40.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.537+0000 7f852468a700 1 -- 172.21.15.3:0/3955531636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f851c0ff700 msgr2=0x7f851c0ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:40.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.537+0000 7f852468a700 1 --2- 172.21.15.3:0/3955531636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f851c0ff700 0x7f851c0ffb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f850c0099c0 tx=0x7f850c009cd0 comp rx=0 tx=0).stop 2023-07-20T01:37:40.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.537+0000 7f852468a700 1 -- 172.21.15.3:0/3955531636 shutdown_connections 2023-07-20T01:37:40.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.537+0000 7f852468a700 1 --2- 172.21.15.3:0/3955531636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f851c1012a0 0x7f851c107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:40.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.537+0000 7f852468a700 1 --2- 172.21.15.3:0/3955531636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f851c0ff700 0x7f851c0ffb20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:40.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.537+0000 7f852468a700 1 --2- 172.21.15.3:0/3955531636 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f851c100900 0x7f851c100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:40.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.537+0000 7f852468a700 1 -- 172.21.15.3:0/3955531636 >> 172.21.15.3:0/3955531636 conn(0x7f851c0fac80 msgr2=0x7f851c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:40.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.537+0000 7f852468a700 1 -- 172.21.15.3:0/3955531636 shutdown_connections 2023-07-20T01:37:40.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.537+0000 7f852468a700 1 -- 172.21.15.3:0/3955531636 wait complete. 2023-07-20T01:37:40.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f852468a700 1 Processor -- start 2023-07-20T01:37:40.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f852468a700 1 -- start start 2023-07-20T01:37:40.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f852468a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f851c0ff700 0x7f851c1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:40.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f852468a700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f851c100900 0x7f851c196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:40.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f852468a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f851c1012a0 0x7f851c19da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:40.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f852468a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c19e060 con 0x7f851c0ff700 2023-07-20T01:37:40.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f852468a700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c19e1a0 con 0x7f851c100900 2023-07-20T01:37:40.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f852468a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c19e330 con 0x7f851c1012a0 2023-07-20T01:37:40.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f8522426700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f851c0ff700 0x7f851c1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:40.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f8522426700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f851c0ff700 0x7f851c1964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41882/0 (socket says 172.21.15.3:41882) 2023-07-20T01:37:40.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f8522426700 1 -- 172.21.15.3:0/2015084972 learned_addr learned my addr 172.21.15.3:0/2015084972 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:37:40.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f8521c25700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f851c100900 0x7f851c196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:40.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f8522c27700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f851c1012a0 0x7f851c19da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:40.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f8522426700 1 -- 172.21.15.3:0/2015084972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f851c1012a0 msgr2=0x7f851c19da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:40.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f8522426700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f851c1012a0 0x7f851c19da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:40.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f8522426700 1 -- 172.21.15.3:0/2015084972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f851c100900 msgr2=0x7f851c196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:40.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f8522426700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f851c100900 0x7f851c196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:40.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f8522426700 1 -- 172.21.15.3:0/2015084972 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0095f0 con 0x7f851c0ff700 2023-07-20T01:37:40.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.538+0000 7f8522426700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f851c0ff700 0x7f851c1964d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f850c0098e0 tx=0x7f850c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:40.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.539+0000 7f85137fe700 1 -- 172.21.15.3:0/2015084972 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c01d070 con 0x7f851c0ff700 2023-07-20T01:37:40.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.539+0000 7f85137fe700 1 -- 172.21.15.3:0/2015084972 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f850c005030 con 0x7f851c0ff700 2023-07-20T01:37:40.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.539+0000 7f85137fe700 1 -- 172.21.15.3:0/2015084972 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c004a90 con 0x7f851c0ff700 2023-07-20T01:37:40.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.539+0000 7f852468a700 1 -- 172.21.15.3:0/2015084972 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851c19e5b0 con 0x7f851c0ff700 2023-07-20T01:37:40.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.539+0000 7f852468a700 1 -- 172.21.15.3:0/2015084972 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851c19e970 con 0x7f851c0ff700 2023-07-20T01:37:40.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.539+0000 7f85137fe700 1 -- 172.21.15.3:0/2015084972 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f850c0051a0 con 0x7f851c0ff700 2023-07-20T01:37:40.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.540+0000 7f85137fe700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f85080389e0 0x7f850803aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:37:40.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.540+0000 7f852468a700 1 -- 172.21.15.3:0/2015084972 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851c10af00 con 0x7f851c0ff700 2023-07-20T01:37:40.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.540+0000 7f85137fe700 1 -- 172.21.15.3:0/2015084972 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f850c04c050 con 0x7f851c0ff700 2023-07-20T01:37:40.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.540+0000 7f8521c25700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f85080389e0 0x7f850803aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:37:40.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.540+0000 7f8521c25700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f85080389e0 0x7f850803aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8518009c60 tx=0x7f8518009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:37:40.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.545+0000 7f85137fe700 1 -- 172.21.15.3:0/2015084972 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f850c026070 con 0x7f851c0ff700 2023-07-20T01:37:40.605 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:40.381+0000 7fe17ac68200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:37:40.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:40 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:40 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:40 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:40 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:40.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:40 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:40.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:40 smithi003 ceph-mon[96422]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:40.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:40.748+0000 7f852468a700 1 -- 172.21.15.3:0/2015084972 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f851c068980 con 0x7f85080389e0 2023-07-20T01:37:41.004 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:40.545+0000 7fc059c68200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T01:37:41.005 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:40.665+0000 7fc059c68200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T01:37:41.041 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:40.684+0000 7fe17ac68200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T01:37:41.042 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:40.924+0000 7fe17ac68200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T01:37:41.042 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:41.071+0000 7fe17ac68200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T01:37:41.505 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:41.188+0000 7fc059c68200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T01:37:41.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:37:41.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:37:41.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:37:41.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:41.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.508 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.508 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:41.508 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:41.508 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.509 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:41.509 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:41.509 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:41.509 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:37:41.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:37:41.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:37:41.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:41.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:41.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:41.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:41.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:41.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:41.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.605 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:41.336+0000 7fe17ac68200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T01:37:41.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:37:41.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:37:41.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:37:41.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:41.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:41.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:41.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:41.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:41.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:41.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:41.607 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:42.255 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:41 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:41.883+0000 7fc059c68200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T01:37:42.255 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:42.075+0000 7fc059c68200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T01:37:42.304 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:42.025+0000 7fe17ac68200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T01:37:42.304 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:42.153+0000 7fe17ac68200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T01:37:42.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:42 smithi086 ceph-mon[98138]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:42.605 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:42.347+0000 7fe17ac68200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T01:37:42.605 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:42.459+0000 7fe17ac68200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T01:37:42.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:42 smithi003 ceph-mon[96422]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:42.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:42 smithi043 ceph-mon[97823]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:43.105 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:42.672+0000 7fe17ac68200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T01:37:43.305 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:43.037+0000 7fc059c68200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:37:43.306 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:43.146+0000 7fc059c68200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T01:37:43.605 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:37:43 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z[100316]: 2023-07-20T01:37:43.544+0000 7fe17ac68200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T01:37:43.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:43 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3306915032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0073d11-a48f-415c-babd-5672ef2585b8"}]: dispatch 2023-07-20T01:37:43.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:43 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3306915032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0073d11-a48f-415c-babd-5672ef2585b8"}]': finished 2023-07-20T01:37:43.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:43 smithi086 ceph-mon[98138]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T01:37:43.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:43 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:37:43.679 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:43.364+0000 7fc059c68200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:37:43.679 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:43 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3306915032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0073d11-a48f-415c-babd-5672ef2585b8"}]: dispatch 2023-07-20T01:37:43.679 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:43 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3306915032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0073d11-a48f-415c-babd-5672ef2585b8"}]': finished 2023-07-20T01:37:43.679 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:43 smithi043 ceph-mon[97823]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T01:37:43.679 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:43 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:37:43.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:43 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3306915032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0073d11-a48f-415c-babd-5672ef2585b8"}]: dispatch 2023-07-20T01:37:43.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:43 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3306915032' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0073d11-a48f-415c-babd-5672ef2585b8"}]': finished 2023-07-20T01:37:43.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:43 smithi003 ceph-mon[96422]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T01:37:43.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:43 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:37:43.933 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:43.709+0000 7fc059c68200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T01:37:44.255 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:43.964+0000 7fc059c68200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T01:37:44.255 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:44 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:44.144+0000 7fc059c68200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T01:37:44.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:44.347+0000 7f85137fe700 1 -- 172.21.15.3:0/2015084972 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 14) v1 ==== 90598+0+0 (secure 0 0 0) 0x7f850c005480 con 0x7f851c0ff700 2023-07-20T01:37:44.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:44 smithi086 ceph-mon[98138]: Standby manager daemon z started 2023-07-20T01:37:44.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:44 smithi086 ceph-mon[98138]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T01:37:44.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:44 smithi086 ceph-mon[98138]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:37:44.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:44 smithi086 ceph-mon[98138]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T01:37:44.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:44 smithi086 ceph-mon[98138]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:37:44.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:44 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3415854710' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:37:44.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:44 smithi086 ceph-mon[98138]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:44.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:44 smithi003 ceph-mon[96422]: Standby manager daemon z started 2023-07-20T01:37:44.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:44 smithi003 ceph-mon[96422]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T01:37:44.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:44 smithi003 ceph-mon[96422]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:37:44.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:44 smithi003 ceph-mon[96422]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T01:37:44.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:44 smithi003 ceph-mon[96422]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:37:44.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:44 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3415854710' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:37:44.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:44 smithi003 ceph-mon[96422]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:44.754 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:44 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:44.468+0000 7fc059c68200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T01:37:44.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:44 smithi043 ceph-mon[97823]: Standby manager daemon z started 2023-07-20T01:37:44.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:44 smithi043 ceph-mon[97823]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T01:37:44.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:44 smithi043 ceph-mon[97823]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:37:44.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:44 smithi043 ceph-mon[97823]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T01:37:44.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:44 smithi043 ceph-mon[97823]: from='mgr.? 172.21.15.86:0/3557422904' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:37:44.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:44 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3415854710' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:37:44.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:44 smithi043 ceph-mon[97823]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:45.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:45 smithi086 ceph-mon[98138]: mgrmap e14: x(active, since 68s), standbys: z 2023-07-20T01:37:45.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:45 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T01:37:45.656 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:45.289+0000 7fc059c68200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T01:37:45.657 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:45.445+0000 7fc059c68200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T01:37:45.657 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:45 smithi043 ceph-mon[97823]: mgrmap e14: x(active, since 68s), standbys: z 2023-07-20T01:37:45.657 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:45 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T01:37:45.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:45 smithi003 ceph-mon[96422]: mgrmap e14: x(active, since 68s), standbys: z 2023-07-20T01:37:45.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:45 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T01:37:46.004 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:45.686+0000 7fc059c68200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T01:37:46.005 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:45.822+0000 7fc059c68200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T01:37:46.320 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:46 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:46.092+0000 7fc059c68200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T01:37:46.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:46 smithi086 ceph-mon[98138]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:46.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:46 smithi003 ceph-mon[96422]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:46.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:46 smithi043 ceph-mon[97823]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:47.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:47.364+0000 7f85137fe700 1 -- 172.21.15.3:0/2015084972 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f850c004bf0 con 0x7f851c0ff700 2023-07-20T01:37:47.505 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:37:47 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y[99306]: 2023-07-20T01:37:47.129+0000 7fc059c68200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T01:37:47.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:47 smithi043 ceph-mon[97823]: Standby manager daemon y started 2023-07-20T01:37:47.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:47 smithi043 ceph-mon[97823]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T01:37:47.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:47 smithi043 ceph-mon[97823]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:37:47.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:47 smithi043 ceph-mon[97823]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T01:37:47.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:47 smithi043 ceph-mon[97823]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:37:47.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:47 smithi086 ceph-mon[98138]: Standby manager daemon y started 2023-07-20T01:37:47.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:47 smithi086 ceph-mon[98138]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T01:37:47.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:47 smithi086 ceph-mon[98138]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:37:47.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:47 smithi086 ceph-mon[98138]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T01:37:47.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:47 smithi086 ceph-mon[98138]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:37:47.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:47 smithi003 ceph-mon[96422]: Standby manager daemon y started 2023-07-20T01:37:47.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:47 smithi003 ceph-mon[96422]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T01:37:47.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:47 smithi003 ceph-mon[96422]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:37:47.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:47 smithi003 ceph-mon[96422]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T01:37:47.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:47 smithi003 ceph-mon[96422]: from='mgr.? 172.21.15.43:0/1216679465' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:37:48.489 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:48 smithi003 ceph-mon[96422]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-07-20T01:37:48.490 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T01:37:48.490 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:48 smithi003 ceph-mon[96422]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:48.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:48 smithi086 ceph-mon[98138]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-07-20T01:37:48.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T01:37:48.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:48 smithi086 ceph-mon[98138]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:48.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:48 smithi043 ceph-mon[97823]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-07-20T01:37:48.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T01:37:48.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:48 smithi043 ceph-mon[97823]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:50.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:49 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T01:37:50.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:49 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:50.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:49 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T01:37:50.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:49 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:50.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:49 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T01:37:50.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:49 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:50.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:50 smithi003 ceph-mon[96422]: Deploying daemon osd.0 on smithi003 2023-07-20T01:37:50.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:50 smithi003 ceph-mon[96422]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:51.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:50 smithi086 ceph-mon[98138]: Deploying daemon osd.0 on smithi003 2023-07-20T01:37:51.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:50 smithi086 ceph-mon[98138]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:51.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:50 smithi043 ceph-mon[97823]: Deploying daemon osd.0 on smithi003 2023-07-20T01:37:51.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:50 smithi043 ceph-mon[97823]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:53.107 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:52 smithi086 ceph-mon[98138]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:53.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:52 smithi003 ceph-mon[96422]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:53.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:52 smithi043 ceph-mon[97823]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:54.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:54 smithi003 ceph-mon[96422]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:54.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:54 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:54.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:54 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:54.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:54 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:55.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:54 smithi086 ceph-mon[98138]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:55.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:54 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:55.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:54 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:55.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:54 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:55.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:54 smithi043 ceph-mon[97823]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:55.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:54 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:55.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:54 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:55.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:54 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:56.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:56 smithi003 ceph-mon[96422]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:56.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:56 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:56.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:56 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:56.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:56 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:56.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:56 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:56.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:56 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:57.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.050+0000 7f85137fe700 1 -- 172.21.15.3:0/2015084972 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f851c068980 con 0x7f85080389e0 2023-07-20T01:37:57.021 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 0 on host 'smithi003' 2023-07-20T01:37:57.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 -- 172.21.15.3:0/2015084972 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f85080389e0 msgr2=0x7f850803aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:57.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f85080389e0 0x7f850803aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8518009c60 tx=0x7f8518009260 comp rx=0 tx=0).stop 2023-07-20T01:37:57.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 -- 172.21.15.3:0/2015084972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f851c0ff700 msgr2=0x7f851c1964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:37:57.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f851c0ff700 0x7f851c1964d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f850c0098e0 tx=0x7f850c009580 comp rx=0 tx=0).stop 2023-07-20T01:37:57.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 -- 172.21.15.3:0/2015084972 shutdown_connections 2023-07-20T01:37:57.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f851c1012a0 0x7f851c19da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:57.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f851c0ff700 0x7f851c1964d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:57.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f85080389e0 0x7f850803aea0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:57.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 --2- 172.21.15.3:0/2015084972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f851c100900 0x7f851c196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:37:57.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 -- 172.21.15.3:0/2015084972 >> 172.21.15.3:0/2015084972 conn(0x7f851c0fac80 msgr2=0x7f851c068270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:37:57.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 -- 172.21.15.3:0/2015084972 shutdown_connections 2023-07-20T01:37:57.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:37:57.052+0000 7f852468a700 1 -- 172.21.15.3:0/2015084972 wait complete. 2023-07-20T01:37:57.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:56 smithi086 ceph-mon[98138]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:57.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:56 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:57.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:56 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:57.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:56 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:57.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:56 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:57.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:56 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:57.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:56 smithi043 ceph-mon[97823]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:57.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:56 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:57.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:56 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:57.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:56 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:37:57.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:56 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:37:57.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:56 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:57.455 DEBUG:teuthology.orchestra.run.smithi003:osd.0> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.0.service 2023-07-20T01:37:57.458 INFO:tasks.cephadm:Deploying osd.1 on smithi043 with /dev/vg_nvme/lv_4... 2023-07-20T01:37:57.459 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T01:37:57.492 INFO:journalctl@ceph.osd.0.smithi003.stdout:-- Logs begin at Thu 2023-07-20 01:20:47 UTC. -- 2023-07-20T01:37:58.014 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:37:58.189 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:37:58 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-0[109510]: 2023-07-20T01:37:58.154+0000 7ff9a4205640 -1 osd.0 0 log_to_monitors true 2023-07-20T01:37:58.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:58 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:58.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:58 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:58.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:58 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:58.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:58 smithi003 ceph-mon[96422]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:58.345 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:58 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:58.346 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:58 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:58.346 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:58 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:58.346 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:58 smithi043 ceph-mon[97823]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:58.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:58 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:58.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:58 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:37:58.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:58 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:37:58.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:58 smithi086 ceph-mon[98138]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:37:59.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:37:59 smithi003 ceph-mon[96422]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T01:37:59.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:37:59 smithi086 ceph-mon[98138]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T01:37:59.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:37:59 smithi043 ceph-mon[97823]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T01:37:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T01:37:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T01:37:59.617 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2023-07-20T01:37:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2023-07-20T01:37:59.617 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0081213 s, 1.3 GB/s 2023-07-20T01:37:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2023-07-20T01:37:59.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi043:vg_nvme/lv_4 2023-07-20T01:38:00.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T01:38:00.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T01:38:00.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:00.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:38:00.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi003 2023-07-20T01:38:00.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:38:00.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi003 to 17173M 2023-07-20T01:38:00.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:00.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:00.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:00 smithi043 ceph-mon[97823]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:38:00.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T01:38:00.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T01:38:00.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:00.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:38:00.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi003 2023-07-20T01:38:00.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:38:00.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi003 to 17173M 2023-07-20T01:38:00.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:00.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:00.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.357 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:00 smithi086 ceph-mon[98138]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:38:00.439 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-0[109510]: 2023-07-20T01:38:00.058+0000 7ff99907c700 -1 osd.0 0 waiting for initial osdmap 2023-07-20T01:38:00.439 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-0[109510]: 2023-07-20T01:38:00.064+0000 7ff99366f700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:38:00.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T01:38:00.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T01:38:00.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:00.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:38:00.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi003 2023-07-20T01:38:00.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:38:00.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi003 to 17173M 2023-07-20T01:38:00.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:00.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:00.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:00.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:00 smithi003 ceph-mon[96422]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:38:00.516 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:38:01.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:01 smithi043 ceph-mon[97823]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:38:01.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:01 smithi043 ceph-mon[97823]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T01:38:01.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:01 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:01.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:01 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:01.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:01 smithi086 ceph-mon[98138]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:38:01.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:01 smithi086 ceph-mon[98138]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T01:38:01.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:01 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:01.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:01 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:01.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:01 smithi003 ceph-mon[96422]: from='osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:38:01.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:01 smithi003 ceph-mon[96422]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T01:38:01.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:01 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:01.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:01 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:01.696 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.724+0000 7f2cbac9d700 1 -- 172.21.15.43:0/1030241918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cb41012a0 msgr2=0x7f2cb4107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:01.697 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.724+0000 7f2cbac9d700 1 --2- 172.21.15.43:0/1030241918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cb41012a0 0x7f2cb4107b50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00b0d0 tx=0x7f2cac00b3e0 comp rx=0 tx=0).stop 2023-07-20T01:38:01.697 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.724+0000 7f2cbac9d700 1 -- 172.21.15.43:0/1030241918 shutdown_connections 2023-07-20T01:38:01.697 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.724+0000 7f2cbac9d700 1 --2- 172.21.15.43:0/1030241918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cb41012a0 0x7f2cb4107b50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:01.697 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.724+0000 7f2cbac9d700 1 --2- 172.21.15.43:0/1030241918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cb40ff700 0x7f2cb40ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:01.697 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.724+0000 7f2cbac9d700 1 --2- 172.21.15.43:0/1030241918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb4100900 0x7f2cb4100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:01.698 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.724+0000 7f2cbac9d700 1 -- 172.21.15.43:0/1030241918 >> 172.21.15.43:0/1030241918 conn(0x7f2cb40fac80 msgr2=0x7f2cb40fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:38:01.698 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.725+0000 7f2cbac9d700 1 -- 172.21.15.43:0/1030241918 shutdown_connections 2023-07-20T01:38:01.698 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.725+0000 7f2cbac9d700 1 -- 172.21.15.43:0/1030241918 wait complete. 2023-07-20T01:38:01.698 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.725+0000 7f2cbac9d700 1 Processor -- start 2023-07-20T01:38:01.698 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.725+0000 7f2cbac9d700 1 -- start start 2023-07-20T01:38:01.698 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.725+0000 7f2cbac9d700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cb40ff700 0x7f2cb41038a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:01.699 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.725+0000 7f2cbac9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb4100900 0x7f2cb4101ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:01.699 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.725+0000 7f2cbac9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cb41012a0 0x7f2cb4102430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:01.699 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.725+0000 7f2cbac9d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb4102bb0 con 0x7f2cb4100900 2023-07-20T01:38:01.699 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.725+0000 7f2cbac9d700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb4102d20 con 0x7f2cb40ff700 2023-07-20T01:38:01.699 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.725+0000 7f2cbac9d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb41a05a0 con 0x7f2cb41012a0 2023-07-20T01:38:01.699 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb8a39700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cb40ff700 0x7f2cb41038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:38:01.699 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb8a39700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cb40ff700 0x7f2cb41038a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48132/0 (socket says 172.21.15.43:48132) 2023-07-20T01:38:01.700 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb8a39700 1 -- 172.21.15.43:0/411033109 learned_addr learned my addr 172.21.15.43:0/411033109 (peer_addr_for_me v2:172.21.15.43:0/0) 2023-07-20T01:38:01.700 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb923a700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cb41012a0 0x7f2cb4102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:38:01.700 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb3fff700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb4100900 0x7f2cb4101ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:38:01.700 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb8a39700 1 -- 172.21.15.43:0/411033109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cb41012a0 msgr2=0x7f2cb4102430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:01.700 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb8a39700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cb41012a0 0x7f2cb4102430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:01.700 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb8a39700 1 -- 172.21.15.43:0/411033109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb4100900 msgr2=0x7f2cb4101ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:01.700 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb8a39700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb4100900 0x7f2cb4101ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:01.700 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb8a39700 1 -- 172.21.15.43:0/411033109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cac009d00 con 0x7f2cb40ff700 2023-07-20T01:38:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb3fff700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb4100900 0x7f2cb4101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:38:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb8a39700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cb40ff700 0x7f2cb41038a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800c970 tx=0x7f2ca800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:38:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb1ffb700 1 -- 172.21.15.43:0/411033109 <== mon.2 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8004220 con 0x7f2cb40ff700 2023-07-20T01:38:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb1ffb700 1 -- 172.21.15.43:0/411033109 <== mon.2 v2:172.21.15.43:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ca8007d60 con 0x7f2cb40ff700 2023-07-20T01:38:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cbac9d700 1 -- 172.21.15.43:0/411033109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb41a0880 con 0x7f2cb40ff700 2023-07-20T01:38:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.726+0000 7f2cb1ffb700 1 -- 172.21.15.43:0/411033109 <== mon.2 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8003c90 con 0x7f2cb40ff700 2023-07-20T01:38:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.727+0000 7f2cbac9d700 1 -- 172.21.15.43:0/411033109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb41a0dd0 con 0x7f2cb40ff700 2023-07-20T01:38:01.702 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.727+0000 7f2cbac9d700 1 -- 172.21.15.43:0/411033109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb41076f0 con 0x7f2cb40ff700 2023-07-20T01:38:01.702 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.728+0000 7f2cb1ffb700 1 -- 172.21.15.43:0/411033109 <== mon.2 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f2ca8004380 con 0x7f2cb40ff700 2023-07-20T01:38:01.702 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.729+0000 7f2cb1ffb700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ca40a1140 0x7f2ca40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.729+0000 7f2cb1ffb700 1 -- 172.21.15.43:0/411033109 <== mon.2 v2:172.21.15.43:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f2ca80ca920 con 0x7f2cb40ff700 2023-07-20T01:38:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.730+0000 7f2cb3fff700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ca40a1140 0x7f2ca40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:38:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.730+0000 7f2cb3fff700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ca40a1140 0x7f2ca40a3600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0005fd0 tx=0x7f2ca0005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:38:01.704 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.734+0000 7f2cb1ffb700 1 -- 172.21.15.43:0/411033109 <== mon.2 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2ca8095ae0 con 0x7f2cb40ff700 2023-07-20T01:38:01.906 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:01.937+0000 7f2cbac9d700 1 -- 172.21.15.43:0/411033109 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cb4060bb0 con 0x7f2ca40a1140 2023-07-20T01:38:02.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:02 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:38:02.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:02 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:38:02.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:02 smithi086 ceph-mon[98138]: osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053] boot 2023-07-20T01:38:02.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:02 smithi086 ceph-mon[98138]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T01:38:02.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:02 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:02.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:02 smithi086 ceph-mon[98138]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:38:02.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:02 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:38:02.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:02 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:38:02.356 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:02 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:02.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:02 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:38:02.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:02 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:38:02.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:02 smithi003 ceph-mon[96422]: osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053] boot 2023-07-20T01:38:02.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:02 smithi003 ceph-mon[96422]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T01:38:02.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:02 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:02.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:02 smithi003 ceph-mon[96422]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:38:02.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:02 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:38:02.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:02 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:38:02.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:02 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:02.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:02 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:38:02.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:02 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:38:02.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:02 smithi043 ceph-mon[97823]: osd.0 [v2:172.21.15.3:6802/1627689053,v1:172.21.15.3:6803/1627689053] boot 2023-07-20T01:38:02.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:02 smithi043 ceph-mon[97823]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T01:38:02.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:02 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:38:02.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:02 smithi043 ceph-mon[97823]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:38:02.506 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:02 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:38:02.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:02 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:38:02.507 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:02 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:03.120 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:03 smithi043 ceph-mon[97823]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:38:03.120 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:03 smithi043 ceph-mon[97823]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T01:38:03.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:03 smithi086 ceph-mon[98138]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:38:03.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:03 smithi086 ceph-mon[98138]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T01:38:03.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:03 smithi003 ceph-mon[96422]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:38:03.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:03 smithi003 ceph-mon[96422]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T01:38:04.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:04 smithi086 ceph-mon[98138]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:04.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:04 smithi003 ceph-mon[96422]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:04.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:04 smithi043 ceph-mon[97823]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:05.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:05 smithi086 ceph-mon[98138]: from='client.? 172.21.15.43:0/3535625462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f60b3d-e08c-4a41-b480-c3d30348eacc"}]: dispatch 2023-07-20T01:38:05.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:05 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f60b3d-e08c-4a41-b480-c3d30348eacc"}]: dispatch 2023-07-20T01:38:05.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:05 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91f60b3d-e08c-4a41-b480-c3d30348eacc"}]': finished 2023-07-20T01:38:05.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:05 smithi086 ceph-mon[98138]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T01:38:05.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:05.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:05 smithi003 ceph-mon[96422]: from='client.? 172.21.15.43:0/3535625462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f60b3d-e08c-4a41-b480-c3d30348eacc"}]: dispatch 2023-07-20T01:38:05.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:05 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f60b3d-e08c-4a41-b480-c3d30348eacc"}]: dispatch 2023-07-20T01:38:05.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:05 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91f60b3d-e08c-4a41-b480-c3d30348eacc"}]': finished 2023-07-20T01:38:05.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:05 smithi003 ceph-mon[96422]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T01:38:05.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:05.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:05 smithi043 ceph-mon[97823]: from='client.? 172.21.15.43:0/3535625462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f60b3d-e08c-4a41-b480-c3d30348eacc"}]: dispatch 2023-07-20T01:38:05.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:05 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f60b3d-e08c-4a41-b480-c3d30348eacc"}]: dispatch 2023-07-20T01:38:05.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:05 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91f60b3d-e08c-4a41-b480-c3d30348eacc"}]': finished 2023-07-20T01:38:05.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:05 smithi043 ceph-mon[97823]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T01:38:05.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:06.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:06 smithi086 ceph-mon[98138]: from='client.? 172.21.15.43:0/2215877309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:38:06.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:06 smithi086 ceph-mon[98138]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:06.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:06 smithi003 ceph-mon[96422]: from='client.? 172.21.15.43:0/2215877309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:38:06.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:06 smithi003 ceph-mon[96422]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:06.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:06 smithi043 ceph-mon[97823]: from='client.? 172.21.15.43:0/2215877309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:38:06.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:06 smithi043 ceph-mon[97823]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:08.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:08 smithi043 ceph-mon[97823]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:08.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:08 smithi086 ceph-mon[98138]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:08.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:08 smithi003 ceph-mon[96422]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:10.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:09 smithi043 ceph-mon[97823]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:10.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:09 smithi086 ceph-mon[98138]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:10.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:09 smithi003 ceph-mon[96422]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:11.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:10 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T01:38:11.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:10 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:11.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:10 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T01:38:11.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:10 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:11.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:10 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T01:38:11.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:10 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:12.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:11 smithi043 ceph-mon[97823]: Deploying daemon osd.1 on smithi043 2023-07-20T01:38:12.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:11 smithi043 ceph-mon[97823]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:12.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:11 smithi003 ceph-mon[96422]: Deploying daemon osd.1 on smithi043 2023-07-20T01:38:12.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:11 smithi003 ceph-mon[96422]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:12.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:11 smithi086 ceph-mon[98138]: Deploying daemon osd.1 on smithi043 2023-07-20T01:38:12.355 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:11 smithi086 ceph-mon[98138]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:14.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:13 smithi086 ceph-mon[98138]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:14.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:13 smithi003 ceph-mon[96422]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:14.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:13 smithi043 ceph-mon[97823]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:16.475 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:16 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:16.476 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:16 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:16.476 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:16 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:16.476 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:16 smithi043 ceph-mon[97823]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:16.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:16 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:16.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:16 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:16.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:16 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:16.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:16 smithi086 ceph-mon[98138]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:16.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:16 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:16.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:16 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:16.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:16 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:16.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:16 smithi003 ceph-mon[96422]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.101+0000 7f2cb1ffb700 1 -- 172.21.15.43:0/411033109 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2cb4060bb0 con 0x7f2ca40a1140 2023-07-20T01:38:18.072 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 1 on host 'smithi043' 2023-07-20T01:38:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.103+0000 7f2cbac9d700 1 -- 172.21.15.43:0/411033109 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ca40a1140 msgr2=0x7f2ca40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.103+0000 7f2cbac9d700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ca40a1140 0x7f2ca40a3600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0005fd0 tx=0x7f2ca0005e30 comp rx=0 tx=0).stop 2023-07-20T01:38:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.104+0000 7f2cbac9d700 1 -- 172.21.15.43:0/411033109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cb40ff700 msgr2=0x7f2cb41038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.104+0000 7f2cbac9d700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cb40ff700 0x7f2cb41038a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800c970 tx=0x7f2ca800cc80 comp rx=0 tx=0).stop 2023-07-20T01:38:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.104+0000 7f2cbac9d700 1 -- 172.21.15.43:0/411033109 shutdown_connections 2023-07-20T01:38:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.104+0000 7f2cbac9d700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cb41012a0 0x7f2cb4102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.104+0000 7f2cbac9d700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ca40a1140 0x7f2ca40a3600 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.104+0000 7f2cbac9d700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cb40ff700 0x7f2cb41038a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.104+0000 7f2cbac9d700 1 --2- 172.21.15.43:0/411033109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb4100900 0x7f2cb4101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.104+0000 7f2cbac9d700 1 -- 172.21.15.43:0/411033109 >> 172.21.15.43:0/411033109 conn(0x7f2cb40fac80 msgr2=0x7f2cb40fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:38:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.104+0000 7f2cbac9d700 1 -- 172.21.15.43:0/411033109 shutdown_connections 2023-07-20T01:38:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:38:18.104+0000 7f2cbac9d700 1 -- 172.21.15.43:0/411033109 wait complete. 2023-07-20T01:38:18.506 DEBUG:teuthology.orchestra.run.smithi043:osd.1> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.1.service 2023-07-20T01:38:18.508 INFO:tasks.cephadm:Deploying osd.2 on smithi086 with /dev/vg_nvme/lv_4... 2023-07-20T01:38:18.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T01:38:18.544 INFO:journalctl@ceph.osd.1.smithi043.stdout:-- Logs begin at Thu 2023-07-20 01:20:44 UTC. -- 2023-07-20T01:38:18.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:18.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:18.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:18 smithi043 ceph-mon[97823]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:18.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:18.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:18.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:18.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:18 smithi086 ceph-mon[98138]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:18.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:18.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:18.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:18.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:18 smithi003 ceph-mon[96422]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:18.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:18.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:19.067 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:38:19.317 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:38:19 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-1[102965]: 2023-07-20T01:38:19.275+0000 7f87a9399640 -1 osd.1 0 log_to_monitors true 2023-07-20T01:38:19.515 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:19 smithi086 ceph-mon[98138]: from='osd.1 [v2:172.21.15.43:6800/3632963249,v1:172.21.15.43:6801/3632963249]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:38:19.516 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:19 smithi086 ceph-mon[98138]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:38:19.568 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:19 smithi043 ceph-mon[97823]: from='osd.1 [v2:172.21.15.43:6800/3632963249,v1:172.21.15.43:6801/3632963249]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:38:19.568 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:19 smithi043 ceph-mon[97823]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:38:19.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:19 smithi003 ceph-mon[96422]: from='osd.1 [v2:172.21.15.43:6800/3632963249,v1:172.21.15.43:6801/3632963249]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:38:19.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:19 smithi003 ceph-mon[96422]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:38:20.395 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T01:38:20.396 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T01:38:20.396 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2023-07-20T01:38:20.396 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2023-07-20T01:38:20.396 INFO:teuthology.orchestra.run.smithi086.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148996 s, 704 MB/s 2023-07-20T01:38:20.396 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2023-07-20T01:38:20.754 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:38:20 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-1[102965]: 2023-07-20T01:38:20.513+0000 7f879e210700 -1 osd.1 0 waiting for initial osdmap 2023-07-20T01:38:20.755 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:38:20 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-1[102965]: 2023-07-20T01:38:20.516+0000 7f879a006700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:38:20.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:20 smithi043 ceph-mon[97823]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T01:38:20.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:20 smithi043 ceph-mon[97823]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T01:38:20.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:20 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:20.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:20 smithi043 ceph-mon[97823]: from='osd.1 [v2:172.21.15.43:6800/3632963249,v1:172.21.15.43:6801/3632963249]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:38:20.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:20 smithi043 ceph-mon[97823]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:38:20.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:20 smithi043 ceph-mon[97823]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:20.794 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:20 smithi086 ceph-mon[98138]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T01:38:20.795 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:20 smithi086 ceph-mon[98138]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T01:38:20.795 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:20 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:20.795 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:20 smithi086 ceph-mon[98138]: from='osd.1 [v2:172.21.15.43:6800/3632963249,v1:172.21.15.43:6801/3632963249]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:38:20.795 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:20 smithi086 ceph-mon[98138]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:38:20.795 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:20 smithi086 ceph-mon[98138]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:20.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi086:vg_nvme/lv_4 2023-07-20T01:38:20.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:20 smithi003 ceph-mon[96422]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T01:38:20.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:20 smithi003 ceph-mon[96422]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T01:38:20.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:20 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:20.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:20 smithi003 ceph-mon[96422]: from='osd.1 [v2:172.21.15.43:6800/3632963249,v1:172.21.15.43:6801/3632963249]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:38:20.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:20 smithi003 ceph-mon[96422]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:38:20.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:20 smithi003 ceph-mon[96422]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:38:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:38:21.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:38:21.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T01:38:21.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:21.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:21.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi043 2023-07-20T01:38:21.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:21.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:21.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:38:21.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi043 to 17173M 2023-07-20T01:38:21.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:21.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:21.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:21.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:21 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:21.815 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:38:21.816 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T01:38:21.816 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:21.816 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:21.816 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi043 2023-07-20T01:38:21.816 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:21.816 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:21.817 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:38:21.817 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi043 to 17173M 2023-07-20T01:38:21.817 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:21.817 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:21.817 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:21.817 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:21 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:21.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:38:21.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T01:38:21.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:21.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:21.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi043 2023-07-20T01:38:21.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:21.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:21.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:38:21.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi043 to 17173M 2023-07-20T01:38:21.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:21.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:21.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:21.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:21 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:22.436 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.464+0000 7f0b33e37700 1 -- 172.21.15.86:0/891400790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b2c101a30 msgr2=0x7f0b2c101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:22.436 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.464+0000 7f0b33e37700 1 --2- 172.21.15.86:0/891400790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b2c101a30 0x7f0b2c101e90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c009990 tx=0x7f0b1c009ca0 comp rx=0 tx=0).stop 2023-07-20T01:38:22.436 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.464+0000 7f0b33e37700 1 -- 172.21.15.86:0/891400790 shutdown_connections 2023-07-20T01:38:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 --2- 172.21.15.86:0/891400790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b2c1023d0 0x7f0b2c108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 --2- 172.21.15.86:0/891400790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c100830 0x7f0b2c100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 --2- 172.21.15.86:0/891400790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b2c101a30 0x7f0b2c101e90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 -- 172.21.15.86:0/891400790 >> 172.21.15.86:0/891400790 conn(0x7f0b2c0fbdf0 msgr2=0x7f0b2c0fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:38:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 -- 172.21.15.86:0/891400790 shutdown_connections 2023-07-20T01:38:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 -- 172.21.15.86:0/891400790 wait complete. 2023-07-20T01:38:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 Processor -- start 2023-07-20T01:38:22.438 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 -- start start 2023-07-20T01:38:22.438 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b2c100830 0x7f0b2c196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:22.438 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b2c101a30 0x7f0b2c196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:22.438 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c1023d0 0x7f0b2c19dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:22.438 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c19e0a0 con 0x7f0b2c101a30 2023-07-20T01:38:22.438 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c19e1e0 con 0x7f0b2c1023d0 2023-07-20T01:38:22.438 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b33e37700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c19e370 con 0x7f0b2c100830 2023-07-20T01:38:22.439 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b31bd3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b2c100830 0x7f0b2c196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:38:22.439 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.465+0000 7f0b31bd3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b2c100830 0x7f0b2c196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58438/0 (socket says 172.21.15.86:58438) 2023-07-20T01:38:22.439 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b31bd3700 1 -- 172.21.15.86:0/1334975720 learned_addr learned my addr 172.21.15.86:0/1334975720 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-07-20T01:38:22.439 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b323d4700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c1023d0 0x7f0b2c19dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:38:22.439 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b31bd3700 1 -- 172.21.15.86:0/1334975720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c1023d0 msgr2=0x7f0b2c19dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:22.439 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b31bd3700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c1023d0 0x7f0b2c19dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:22.439 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b31bd3700 1 -- 172.21.15.86:0/1334975720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b2c101a30 msgr2=0x7f0b2c196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:22.439 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b31bd3700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b2c101a30 0x7f0b2c196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:22.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b31bd3700 1 -- 172.21.15.86:0/1334975720 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c0095f0 con 0x7f0b2c100830 2023-07-20T01:38:22.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b323d4700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c1023d0 0x7f0b2c19dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:38:22.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b31bd3700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b2c100830 0x7f0b2c196510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800d9b0 tx=0x7f0b2800dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:38:22.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b22ffd700 1 -- 172.21.15.86:0/1334975720 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2800be80 con 0x7f0b2c100830 2023-07-20T01:38:22.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b22ffd700 1 -- 172.21.15.86:0/1334975720 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b280044e0 con 0x7f0b2c100830 2023-07-20T01:38:22.441 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b22ffd700 1 -- 172.21.15.86:0/1334975720 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b28003b40 con 0x7f0b2c100830 2023-07-20T01:38:22.441 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b33e37700 1 -- 172.21.15.86:0/1334975720 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c19e650 con 0x7f0b2c100830 2023-07-20T01:38:22.441 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.466+0000 7f0b33e37700 1 -- 172.21.15.86:0/1334975720 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c19eb50 con 0x7f0b2c100830 2023-07-20T01:38:22.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.467+0000 7f0b33e37700 1 -- 172.21.15.86:0/1334975720 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c107e30 con 0x7f0b2c100830 2023-07-20T01:38:22.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.468+0000 7f0b22ffd700 1 -- 172.21.15.86:0/1334975720 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f0b28009940 con 0x7f0b2c100830 2023-07-20T01:38:22.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.469+0000 7f0b22ffd700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0b180a11c0 0x7f0b180a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:22.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.469+0000 7f0b22ffd700 1 -- 172.21.15.86:0/1334975720 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f0b280cb160 con 0x7f0b2c100830 2023-07-20T01:38:22.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.469+0000 7f0b313d2700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0b180a11c0 0x7f0b180a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:38:22.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.469+0000 7f0b313d2700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0b180a11c0 0x7f0b180a3680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c0095c0 tx=0x7f0b1c009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:38:22.443 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.472+0000 7f0b22ffd700 1 -- 172.21.15.86:0/1334975720 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0b280960e0 con 0x7f0b2c100830 2023-07-20T01:38:22.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:22 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:38:22.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:22 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:38:22.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:22 smithi086 ceph-mon[98138]: osd.1 [v2:172.21.15.43:6800/3632963249,v1:172.21.15.43:6801/3632963249] boot 2023-07-20T01:38:22.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:22 smithi086 ceph-mon[98138]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T01:38:22.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:22 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:22.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:22 smithi086 ceph-mon[98138]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:22.638 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:22.668+0000 7f0b33e37700 1 -- 172.21.15.86:0/1334975720 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b2c060bb0 con 0x7f0b180a11c0 2023-07-20T01:38:22.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:22 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:38:22.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:22 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:38:22.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:22 smithi043 ceph-mon[97823]: osd.1 [v2:172.21.15.43:6800/3632963249,v1:172.21.15.43:6801/3632963249] boot 2023-07-20T01:38:22.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:22 smithi043 ceph-mon[97823]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T01:38:22.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:22.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:22 smithi043 ceph-mon[97823]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:22.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:22 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:38:22.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:22 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:38:22.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:22 smithi003 ceph-mon[96422]: osd.1 [v2:172.21.15.43:6800/3632963249,v1:172.21.15.43:6801/3632963249] boot 2023-07-20T01:38:22.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:22 smithi003 ceph-mon[96422]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T01:38:22.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:22 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:38:22.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:22 smithi003 ceph-mon[96422]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:23.706 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:23 smithi086 ceph-mon[98138]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T01:38:23.706 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:23 smithi086 ceph-mon[98138]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:38:23.707 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:38:23.707 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:38:23.707 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:23.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:23 smithi043 ceph-mon[97823]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T01:38:23.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:23 smithi043 ceph-mon[97823]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:38:23.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:38:23.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:38:23.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:23.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:23 smithi003 ceph-mon[96422]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T01:38:23.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:23 smithi003 ceph-mon[96422]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:38:23.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:38:23.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:38:23.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:24.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:24 smithi043 ceph-mon[97823]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:24.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:24 smithi086 ceph-mon[98138]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:24.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:24 smithi003 ceph-mon[96422]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:25.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:25 smithi043 ceph-mon[97823]: from='client.? 172.21.15.86:0/2364840411' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11808f4b-c50a-44f4-8581-3fc226653b39"}]: dispatch 2023-07-20T01:38:25.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:25 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11808f4b-c50a-44f4-8581-3fc226653b39"}]: dispatch 2023-07-20T01:38:25.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:25 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11808f4b-c50a-44f4-8581-3fc226653b39"}]': finished 2023-07-20T01:38:25.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:25 smithi043 ceph-mon[97823]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T01:38:25.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:25 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:25.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:25 smithi086 ceph-mon[98138]: from='client.? 172.21.15.86:0/2364840411' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11808f4b-c50a-44f4-8581-3fc226653b39"}]: dispatch 2023-07-20T01:38:25.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:25 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11808f4b-c50a-44f4-8581-3fc226653b39"}]: dispatch 2023-07-20T01:38:25.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:25 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11808f4b-c50a-44f4-8581-3fc226653b39"}]': finished 2023-07-20T01:38:25.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:25 smithi086 ceph-mon[98138]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T01:38:25.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:25 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:25.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:25 smithi003 ceph-mon[96422]: from='client.? 172.21.15.86:0/2364840411' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11808f4b-c50a-44f4-8581-3fc226653b39"}]: dispatch 2023-07-20T01:38:25.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:25 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11808f4b-c50a-44f4-8581-3fc226653b39"}]: dispatch 2023-07-20T01:38:25.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:25 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11808f4b-c50a-44f4-8581-3fc226653b39"}]': finished 2023-07-20T01:38:25.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:25 smithi003 ceph-mon[96422]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T01:38:25.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:25 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:26.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:26 smithi086 ceph-mon[98138]: from='client.? 172.21.15.86:0/1362552234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:38:26.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:26 smithi086 ceph-mon[98138]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:26.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:26 smithi003 ceph-mon[96422]: from='client.? 172.21.15.86:0/1362552234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:38:26.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:26 smithi003 ceph-mon[96422]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:27.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:26 smithi043 ceph-mon[97823]: from='client.? 172.21.15.86:0/1362552234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:38:27.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:26 smithi043 ceph-mon[97823]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:28.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:28 smithi086 ceph-mon[98138]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:28.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:28 smithi003 ceph-mon[96422]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:28.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:28 smithi043 ceph-mon[97823]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:30.164 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:29 smithi086 ceph-mon[98138]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:30.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:29 smithi003 ceph-mon[96422]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:30.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:29 smithi043 ceph-mon[97823]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:31.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:31 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T01:38:31.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:31 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:31.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:31 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T01:38:31.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:31 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:32.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:31 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T01:38:32.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:31 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:32.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:32 smithi086 ceph-mon[98138]: Deploying daemon osd.2 on smithi086 2023-07-20T01:38:32.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:32 smithi086 ceph-mon[98138]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:32.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:32 smithi003 ceph-mon[96422]: Deploying daemon osd.2 on smithi086 2023-07-20T01:38:32.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:32 smithi003 ceph-mon[96422]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:33.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:32 smithi043 ceph-mon[97823]: Deploying daemon osd.2 on smithi086 2023-07-20T01:38:33.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:32 smithi043 ceph-mon[97823]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:34.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:33 smithi086 ceph-mon[98138]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:34.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:33 smithi003 ceph-mon[96422]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:34.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:33 smithi043 ceph-mon[97823]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:36.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:36.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:36.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:36.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:36 smithi086 ceph-mon[98138]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:36.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:36.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:36 smithi003 ceph-mon[96422]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:37.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:37.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:37.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:37.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:36 smithi043 ceph-mon[97823]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.429+0000 7f0b22ffd700 1 -- 172.21.15.86:0/1334975720 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0b2c060bb0 con 0x7f0b180a11c0 2023-07-20T01:38:38.400 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 2 on host 'smithi086' 2023-07-20T01:38:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 -- 172.21.15.86:0/1334975720 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0b180a11c0 msgr2=0x7f0b180a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0b180a11c0 0x7f0b180a3680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c0095c0 tx=0x7f0b1c009510 comp rx=0 tx=0).stop 2023-07-20T01:38:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 -- 172.21.15.86:0/1334975720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b2c100830 msgr2=0x7f0b2c196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b2c100830 0x7f0b2c196510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800d9b0 tx=0x7f0b2800dd70 comp rx=0 tx=0).stop 2023-07-20T01:38:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 -- 172.21.15.86:0/1334975720 shutdown_connections 2023-07-20T01:38:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0b180a11c0 0x7f0b180a3680 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c1023d0 0x7f0b2c19dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b2c100830 0x7f0b2c196510 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 --2- 172.21.15.86:0/1334975720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b2c101a30 0x7f0b2c196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 -- 172.21.15.86:0/1334975720 >> 172.21.15.86:0/1334975720 conn(0x7f0b2c0fbdf0 msgr2=0x7f0b2c104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:38:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 -- 172.21.15.86:0/1334975720 shutdown_connections 2023-07-20T01:38:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:38:38.432+0000 7f0b33e37700 1 -- 172.21.15.86:0/1334975720 wait complete. 2023-07-20T01:38:38.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:38.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:38.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:38.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:38.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:38.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:38 smithi086 ceph-mon[98138]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:38.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:38.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:38.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:38 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:38.746 DEBUG:teuthology.orchestra.run.smithi086:osd.2> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.2.service 2023-07-20T01:38:38.749 INFO:tasks.cephadm:Deploying osd.3 on smithi003 with /dev/vg_nvme/lv_3... 2023-07-20T01:38:38.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T01:38:38.778 INFO:journalctl@ceph.osd.2.smithi086.stdout:-- Logs begin at Thu 2023-07-20 01:20:49 UTC. -- 2023-07-20T01:38:38.785 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:38.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:38.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:38.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:38.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:38.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:38 smithi003 ceph-mon[96422]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:38.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:38.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:38.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:38 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:39.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:39.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:39.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:39.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:39.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:39.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:38 smithi043 ceph-mon[97823]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:39.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:39.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:39.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:38 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:39.328 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:38:39.549 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:38:39 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-2[104035]: 2023-07-20T01:38:39.415+0000 7fce7b80d640 -1 osd.2 0 log_to_monitors true 2023-07-20T01:38:39.830 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:39 smithi003 ceph-mon[96422]: from='osd.2 [v2:172.21.15.86:6800/2141120676,v1:172.21.15.86:6801/2141120676]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:38:39.831 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:39 smithi003 ceph-mon[96422]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:38:39.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:39 smithi086 ceph-mon[98138]: from='osd.2 [v2:172.21.15.86:6800/2141120676,v1:172.21.15.86:6801/2141120676]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:38:39.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:39 smithi086 ceph-mon[98138]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:38:40.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:39 smithi043 ceph-mon[97823]: from='osd.2 [v2:172.21.15.86:6800/2141120676,v1:172.21.15.86:6801/2141120676]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:38:40.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:39 smithi043 ceph-mon[97823]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:38:40.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:40 smithi003 ceph-mon[96422]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T01:38:40.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:40 smithi003 ceph-mon[96422]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T01:38:40.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:40 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:40.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:40 smithi003 ceph-mon[96422]: from='osd.2 [v2:172.21.15.86:6800/2141120676,v1:172.21.15.86:6801/2141120676]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:38:40.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:40 smithi003 ceph-mon[96422]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:38:40.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:40 smithi003 ceph-mon[96422]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:40.800 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:38:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-2[104035]: 2023-07-20T01:38:40.598+0000 7fce70684700 -1 osd.2 0 waiting for initial osdmap 2023-07-20T01:38:40.801 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:38:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-2[104035]: 2023-07-20T01:38:40.601+0000 7fce6cc7b700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:38:40.801 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:40 smithi086 ceph-mon[98138]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T01:38:40.801 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:40 smithi086 ceph-mon[98138]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T01:38:40.801 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:40 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:40.802 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:40 smithi086 ceph-mon[98138]: from='osd.2 [v2:172.21.15.86:6800/2141120676,v1:172.21.15.86:6801/2141120676]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:38:40.802 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:40 smithi086 ceph-mon[98138]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:38:40.802 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:40 smithi086 ceph-mon[98138]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:40.835 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T01:38:40.836 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T01:38:40.836 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2023-07-20T01:38:40.836 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2023-07-20T01:38:40.836 INFO:teuthology.orchestra.run.smithi003.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0107743 s, 973 MB/s 2023-07-20T01:38:40.836 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2023-07-20T01:38:41.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:40 smithi043 ceph-mon[97823]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T01:38:41.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:40 smithi043 ceph-mon[97823]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T01:38:41.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:40 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:41.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:40 smithi043 ceph-mon[97823]: from='osd.2 [v2:172.21.15.86:6800/2141120676,v1:172.21.15.86:6801/2141120676]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:38:41.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:40 smithi043 ceph-mon[97823]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:38:41.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:40 smithi043 ceph-mon[97823]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:41.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi003:vg_nvme/lv_3 2023-07-20T01:38:41.790 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:38:41.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:41 smithi086 ceph-mon[98138]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:38:41.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:41 smithi086 ceph-mon[98138]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T01:38:41.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:41.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:41.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:41.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:41.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:38:41.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:41.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:41.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:41.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:41 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:41.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:41 smithi003 ceph-mon[96422]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:38:41.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:41 smithi003 ceph-mon[96422]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T01:38:41.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:41.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:41.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:41.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:41.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:38:41.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:41.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:41.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:41.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:41 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:42.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:41 smithi043 ceph-mon[97823]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:38:42.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:41 smithi043 ceph-mon[97823]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T01:38:42.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:42.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:42.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:42.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:42.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:38:42.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:42.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:42.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:38:42.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:41 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:42.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:42 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:38:42.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:42 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:38:42.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:42 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi086 2023-07-20T01:38:42.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:42 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi086 to 17168M 2023-07-20T01:38:42.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:42 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:42.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:42 smithi003 ceph-mon[96422]: osd.2 [v2:172.21.15.86:6800/2141120676,v1:172.21.15.86:6801/2141120676] boot 2023-07-20T01:38:42.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:42 smithi003 ceph-mon[96422]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T01:38:42.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:42 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:42.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:42 smithi003 ceph-mon[96422]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:42.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:42 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:38:42.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:42 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:38:42.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:42 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:38:42.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:42 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi086 2023-07-20T01:38:42.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:42 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi086 to 17168M 2023-07-20T01:38:42.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:42 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:42.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:42 smithi086 ceph-mon[98138]: osd.2 [v2:172.21.15.86:6800/2141120676,v1:172.21.15.86:6801/2141120676] boot 2023-07-20T01:38:42.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:42 smithi086 ceph-mon[98138]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T01:38:42.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:42 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:42.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:42 smithi086 ceph-mon[98138]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:42.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:42 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:38:43.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:42 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:38:43.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:42 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:38:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:42 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi086 2023-07-20T01:38:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:42 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi086 to 17168M 2023-07-20T01:38:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:42 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:42 smithi043 ceph-mon[97823]: osd.2 [v2:172.21.15.86:6800/2141120676,v1:172.21.15.86:6801/2141120676] boot 2023-07-20T01:38:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:42 smithi043 ceph-mon[97823]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T01:38:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:42 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:38:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:42 smithi043 ceph-mon[97823]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:38:43.006 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:42 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:38:43.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.036+0000 7f2af46a3700 1 -- 172.21.15.3:0/1731949108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aec1012a0 msgr2=0x7f2aec107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:43.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.036+0000 7f2af46a3700 1 --2- 172.21.15.3:0/1731949108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aec1012a0 0x7f2aec107b50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2ae400b0d0 tx=0x7f2ae400b3e0 comp rx=0 tx=0).stop 2023-07-20T01:38:43.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.036+0000 7f2af46a3700 1 -- 172.21.15.3:0/1731949108 shutdown_connections 2023-07-20T01:38:43.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.036+0000 7f2af46a3700 1 --2- 172.21.15.3:0/1731949108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aec1012a0 0x7f2aec107b50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:43.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.036+0000 7f2af46a3700 1 --2- 172.21.15.3:0/1731949108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2aec0ff700 0x7f2aec0ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:43.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.036+0000 7f2af46a3700 1 --2- 172.21.15.3:0/1731949108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec100900 0x7f2aec100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:43.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.036+0000 7f2af46a3700 1 -- 172.21.15.3:0/1731949108 >> 172.21.15.3:0/1731949108 conn(0x7f2aec0fac80 msgr2=0x7f2aec0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:38:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.036+0000 7f2af46a3700 1 -- 172.21.15.3:0/1731949108 shutdown_connections 2023-07-20T01:38:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.036+0000 7f2af46a3700 1 -- 172.21.15.3:0/1731949108 wait complete. 2023-07-20T01:38:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.036+0000 7f2af46a3700 1 Processor -- start 2023-07-20T01:38:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.036+0000 7f2af46a3700 1 -- start start 2023-07-20T01:38:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af46a3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aec0ff700 0x7f2aec1965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af46a3700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2aec100900 0x7f2aec196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af46a3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec1012a0 0x7f2aec19db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af46a3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec19e140 con 0x7f2aec0ff700 2023-07-20T01:38:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af46a3700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec19e280 con 0x7f2aec100900 2023-07-20T01:38:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af46a3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec19e410 con 0x7f2aec1012a0 2023-07-20T01:38:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af243f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aec0ff700 0x7f2aec1965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:38:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af243f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aec0ff700 0x7f2aec1965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41952/0 (socket says 172.21.15.3:41952) 2023-07-20T01:38:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af243f700 1 -- 172.21.15.3:0/2715556982 learned_addr learned my addr 172.21.15.3:0/2715556982 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:38:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af1c3e700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2aec100900 0x7f2aec196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:38:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af2c40700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec1012a0 0x7f2aec19db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:38:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af243f700 1 -- 172.21.15.3:0/2715556982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec1012a0 msgr2=0x7f2aec19db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af243f700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec1012a0 0x7f2aec19db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af243f700 1 -- 172.21.15.3:0/2715556982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2aec100900 msgr2=0x7f2aec196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:38:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af243f700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2aec100900 0x7f2aec196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:38:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af243f700 1 -- 172.21.15.3:0/2715556982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae4009d00 con 0x7f2aec0ff700 2023-07-20T01:38:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af1c3e700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2aec100900 0x7f2aec196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:38:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af2c40700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec1012a0 0x7f2aec19db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:38:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af243f700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aec0ff700 0x7f2aec1965b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800e9a0 tx=0x7f2ae800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:38:43.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2ae37fe700 1 -- 172.21.15.3:0/2715556982 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8009790 con 0x7f2aec0ff700 2023-07-20T01:38:43.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2ae37fe700 1 -- 172.21.15.3:0/2715556982 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ae80044e0 con 0x7f2aec0ff700 2023-07-20T01:38:43.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2af46a3700 1 -- 172.21.15.3:0/2715556982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aec19e6f0 con 0x7f2aec0ff700 2023-07-20T01:38:43.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.037+0000 7f2ae37fe700 1 -- 172.21.15.3:0/2715556982 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8003d70 con 0x7f2aec0ff700 2023-07-20T01:38:43.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.038+0000 7f2af46a3700 1 -- 172.21.15.3:0/2715556982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aec19ec40 con 0x7f2aec0ff700 2023-07-20T01:38:43.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.039+0000 7f2af46a3700 1 -- 172.21.15.3:0/2715556982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aec106c80 con 0x7f2aec0ff700 2023-07-20T01:38:43.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.040+0000 7f2ae37fe700 1 -- 172.21.15.3:0/2715556982 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f2ae8003ef0 con 0x7f2aec0ff700 2023-07-20T01:38:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.041+0000 7f2ae37fe700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ad80a1140 0x7f2ad80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:38:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.041+0000 7f2af1c3e700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ad80a1140 0x7f2ad80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:38:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.041+0000 7f2ae37fe700 1 -- 172.21.15.3:0/2715556982 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7f2ae8014070 con 0x7f2aec0ff700 2023-07-20T01:38:43.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.041+0000 7f2af1c3e700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ad80a1140 0x7f2ad80a3600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2adc009cb0 tx=0x7f2adc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:38:43.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.044+0000 7f2ae37fe700 1 -- 172.21.15.3:0/2715556982 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2ae80963d0 con 0x7f2aec0ff700 2023-07-20T01:38:43.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:43.249+0000 7f2af46a3700 1 -- 172.21.15.3:0/2715556982 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2aec060bb0 con 0x7f2ad80a1140 2023-07-20T01:38:43.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:43 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T01:38:43.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:43 smithi086 ceph-mon[98138]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T01:38:43.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:43 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:38:43.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:43 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:38:43.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:43 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:38:43.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:43 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:43.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:43 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T01:38:43.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:43 smithi003 ceph-mon[96422]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T01:38:43.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:43 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:38:43.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:43 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:38:43.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:43 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:38:43.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:43 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:44.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:43 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T01:38:44.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:43 smithi043 ceph-mon[97823]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T01:38:44.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:43 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:38:44.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:43 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:38:44.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:43 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:38:44.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:43 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:44.414 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:38:44 smithi003 sudo[112546]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:38:44.414 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:38:44 smithi003 sudo[112546]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:44.415 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:38:44 smithi003 sudo[112546]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:44.415 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:38:44 smithi003 sudo[112546]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:44.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:38:44.626+0000 7f2ae37fe700 1 -- 172.21.15.3:0/2715556982 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f2ae80967a0 con 0x7f2aec0ff700 2023-07-20T01:38:44.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:44 smithi003 ceph-mon[96422]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:38:44.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:44 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T01:38:44.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:44 smithi003 ceph-mon[96422]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T01:38:44.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:44 smithi003 ceph-mon[96422]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:44.690 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:38:44 smithi003 sudo[112593]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:38:44.690 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:38:44 smithi003 sudo[112593]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:44.690 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:38:44 smithi003 sudo[112593]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:44.690 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:38:44 smithi003 sudo[112593]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:44.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:44 smithi086 ceph-mon[98138]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:38:44.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:44 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T01:38:44.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:44 smithi086 ceph-mon[98138]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T01:38:44.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:44 smithi086 ceph-mon[98138]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:45.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:44 smithi043 ceph-mon[97823]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:38:45.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:44 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T01:38:45.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:44 smithi043 ceph-mon[97823]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T01:38:45.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:44 smithi043 ceph-mon[97823]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:45.504 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:38:45 smithi043 sudo[105267]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:38:45.504 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:38:45 smithi043 sudo[105267]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:45.504 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:38:45 smithi043 sudo[105267]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:45.505 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:38:45 smithi043 sudo[105267]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:45.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:45 smithi086 ceph-mon[98138]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T01:38:45.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:45 smithi086 ceph-mon[98138]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T01:38:45.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:45 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:45.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:45 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:45.855 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:38:45 smithi086 sudo[106178]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:38:45.855 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:38:45 smithi086 sudo[106178]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:45.855 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:38:45 smithi086 sudo[106178]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:45.855 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:38:45 smithi086 sudo[106178]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:45.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:45 smithi003 ceph-mon[96422]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T01:38:45.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:45 smithi003 ceph-mon[96422]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T01:38:45.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:45 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:45.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:45 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:46.004 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:38:45 smithi043 sudo[105270]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:38:46.004 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:38:45 smithi043 sudo[105270]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:46.004 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:38:45 smithi043 sudo[105270]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:46.005 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:38:45 smithi043 sudo[105270]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:46.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:45 smithi043 ceph-mon[97823]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T01:38:46.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:45 smithi043 ceph-mon[97823]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T01:38:46.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:45 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:46.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:45 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:46.354 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:38:46 smithi086 sudo[106181]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:38:46.354 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:38:46 smithi086 sudo[106181]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:46.355 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:38:46 smithi086 sudo[106181]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:46.355 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:38:46 smithi086 sudo[106181]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:46.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 sudo[112716]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:38:46.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 sudo[112716]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:46.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 sudo[112716]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:46.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 sudo[112716]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:47.213 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:46 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3479183034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45c2d3f6-9f5d-4510-9dad-6d95d044725f"}]: dispatch 2023-07-20T01:38:47.213 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:46 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3479183034' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45c2d3f6-9f5d-4510-9dad-6d95d044725f"}]': finished 2023-07-20T01:38:47.213 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:46 smithi086 ceph-mon[98138]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T01:38:47.214 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:46 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:38:47.214 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:46 smithi086 ceph-mon[98138]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:47.214 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:46 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:47.214 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:46 smithi086 ceph-mon[98138]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:38:47.214 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:47 smithi086 sudo[106184]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:38:47.214 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:47 smithi086 sudo[106184]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:47.214 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:47 smithi086 sudo[106184]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:47.215 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:47 smithi086 sudo[106184]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:47.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:46 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3479183034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45c2d3f6-9f5d-4510-9dad-6d95d044725f"}]: dispatch 2023-07-20T01:38:47.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:46 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3479183034' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45c2d3f6-9f5d-4510-9dad-6d95d044725f"}]': finished 2023-07-20T01:38:47.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:46 smithi043 ceph-mon[97823]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T01:38:47.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:46 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:38:47.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:46 smithi043 ceph-mon[97823]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:47.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:46 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:47.257 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:46 smithi043 ceph-mon[97823]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:38:47.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 sudo[112719]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:38:47.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 sudo[112719]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:47.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 sudo[112719]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:47.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 sudo[112719]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:47.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3479183034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45c2d3f6-9f5d-4510-9dad-6d95d044725f"}]: dispatch 2023-07-20T01:38:47.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3479183034' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45c2d3f6-9f5d-4510-9dad-6d95d044725f"}]': finished 2023-07-20T01:38:47.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 ceph-mon[96422]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T01:38:47.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:38:47.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 ceph-mon[96422]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:47.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:47.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:46 smithi003 ceph-mon[96422]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:38:47.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:47 smithi086 sudo[106187]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:38:47.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:47 smithi086 sudo[106187]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:47.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:47 smithi086 sudo[106187]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:47.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:47 smithi086 sudo[106187]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:48.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:47 smithi043 sudo[105273]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:38:48.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:47 smithi043 sudo[105273]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:48.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:47 smithi043 sudo[105273]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:48.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:47 smithi043 sudo[105273]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:48.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:38:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:38:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:38:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:38:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:38:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:38:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:38:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:38:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3274685679' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:38:48.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:38:48.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:38:48.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:38:48.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:38:48.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:48 smithi003 ceph-mon[96422]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:48.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 sudo[105276]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:38:48.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 sudo[105276]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:38:48.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 sudo[105276]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:38:48.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 sudo[105276]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:38:48.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:38:48.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:38:48.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:38:48.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:38:48.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:38:48.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:38:48.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:38:48.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:38:48.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3274685679' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:38:48.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:38:48.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:38:48.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:38:48.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:38:48.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:48 smithi043 ceph-mon[97823]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:48.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:38:48.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:38:48.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:38:48.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:38:48.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:38:48.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:38:48.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:38:48.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:38:48.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3274685679' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:38:48.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:38:48.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:38:48.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:38:48.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:38:48.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:48 smithi086 ceph-mon[98138]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:49.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:49 smithi003 ceph-mon[96422]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:38:49.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:49 smithi003 ceph-mon[96422]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:38:49.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:49 smithi043 ceph-mon[97823]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:38:49.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:49 smithi043 ceph-mon[97823]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:38:49.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:49 smithi086 ceph-mon[98138]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:38:49.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:49 smithi086 ceph-mon[98138]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:38:50.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:50 smithi003 ceph-mon[96422]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:50.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:50 smithi043 ceph-mon[97823]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:50.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:50 smithi086 ceph-mon[98138]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:38:52.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:51 smithi003 ceph-mon[96422]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-20T01:38:52.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:51 smithi043 ceph-mon[97823]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-20T01:38:52.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:51 smithi086 ceph-mon[98138]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-20T01:38:53.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:53 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T01:38:53.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:53 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:53.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:53 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T01:38:53.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:53 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:53.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:53 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T01:38:53.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:53 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:38:54.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:54 smithi003 ceph-mon[96422]: Deploying daemon osd.3 on smithi003 2023-07-20T01:38:54.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:54 smithi003 ceph-mon[96422]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-20T01:38:54.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:54 smithi043 ceph-mon[97823]: Deploying daemon osd.3 on smithi003 2023-07-20T01:38:54.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:54 smithi043 ceph-mon[97823]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-20T01:38:54.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:54 smithi086 ceph-mon[98138]: Deploying daemon osd.3 on smithi003 2023-07-20T01:38:54.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:54 smithi086 ceph-mon[98138]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-20T01:38:56.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:55 smithi003 ceph-mon[96422]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-20T01:38:56.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:55 smithi043 ceph-mon[97823]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-20T01:38:56.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:55 smithi086 ceph-mon[98138]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-20T01:38:58.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:58 smithi003 ceph-mon[96422]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-20T01:38:58.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:58 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:58.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:58 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:58.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:38:58 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:59.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:58 smithi086 ceph-mon[98138]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-20T01:38:59.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:58 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:59.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:58 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:59.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:38:58 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:38:59.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:58 smithi043 ceph-mon[97823]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-20T01:38:59.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:58 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:59.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:58 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:38:59.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:38:58 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:00.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.508+0000 7f2ae37fe700 1 -- 172.21.15.3:0/2715556982 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2aec060bb0 con 0x7f2ad80a1140 2023-07-20T01:39:00.479 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 3 on host 'smithi003' 2023-07-20T01:39:00.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 -- 172.21.15.3:0/2715556982 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ad80a1140 msgr2=0x7f2ad80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:00.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ad80a1140 0x7f2ad80a3600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2adc009cb0 tx=0x7f2adc009260 comp rx=0 tx=0).stop 2023-07-20T01:39:00.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 -- 172.21.15.3:0/2715556982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aec0ff700 msgr2=0x7f2aec1965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:00.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aec0ff700 0x7f2aec1965b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800e9a0 tx=0x7f2ae800ed60 comp rx=0 tx=0).stop 2023-07-20T01:39:00.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 -- 172.21.15.3:0/2715556982 shutdown_connections 2023-07-20T01:39:00.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec1012a0 0x7f2aec19db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:00.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aec0ff700 0x7f2aec1965b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:00.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2ad80a1140 0x7f2ad80a3600 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:00.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 --2- 172.21.15.3:0/2715556982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2aec100900 0x7f2aec196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:00.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 -- 172.21.15.3:0/2715556982 >> 172.21.15.3:0/2715556982 conn(0x7f2aec0fac80 msgr2=0x7f2aec103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:39:00.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 -- 172.21.15.3:0/2715556982 shutdown_connections 2023-07-20T01:39:00.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:00.511+0000 7f2af46a3700 1 -- 172.21.15.3:0/2715556982 wait complete. 2023-07-20T01:39:00.885 DEBUG:teuthology.orchestra.run.smithi003:osd.3> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.3.service 2023-07-20T01:39:00.887 INFO:tasks.cephadm:Deploying osd.4 on smithi043 with /dev/vg_nvme/lv_3... 2023-07-20T01:39:00.887 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T01:39:00.916 INFO:journalctl@ceph.osd.3.smithi003.stdout:-- Logs begin at Thu 2023-07-20 01:20:47 UTC. -- 2023-07-20T01:39:01.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:00 smithi086 ceph-mon[98138]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T01:39:01.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:00 smithi003 ceph-mon[96422]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T01:39:01.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:00 smithi043 ceph-mon[97823]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T01:39:01.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:01.459 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:39:01.985 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:01 smithi043 ceph-mon[97823]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T01:39:02.049 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:01 smithi003 ceph-mon[96422]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T01:39:02.049 INFO:journalctl@ceph.osd.3.smithi003.stdout:Jul 20 01:39:01 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-3[114976]: 2023-07-20T01:39:01.764+0000 7f563c97d640 -1 osd.3 0 log_to_monitors true 2023-07-20T01:39:02.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:01 smithi086 ceph-mon[98138]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T01:39:03.040 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T01:39:03.040 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T01:39:03.040 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2023-07-20T01:39:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2023-07-20T01:39:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130997 s, 800 MB/s 2023-07-20T01:39:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2023-07-20T01:39:03.080 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:02 smithi043 ceph-mon[97823]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T01:39:03.080 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:02 smithi043 ceph-mon[97823]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T01:39:03.081 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:02 smithi043 ceph-mon[97823]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T01:39:03.081 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:02 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:03.081 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:02 smithi043 ceph-mon[97823]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:39:03.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:02 smithi086 ceph-mon[98138]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T01:39:03.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:02 smithi086 ceph-mon[98138]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T01:39:03.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:02 smithi086 ceph-mon[98138]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T01:39:03.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:02 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:03.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:02 smithi086 ceph-mon[98138]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:39:03.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:02 smithi003 ceph-mon[96422]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-20T01:39:03.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:02 smithi003 ceph-mon[96422]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T01:39:03.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:02 smithi003 ceph-mon[96422]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T01:39:03.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:02 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:03.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:02 smithi003 ceph-mon[96422]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:39:03.190 INFO:journalctl@ceph.osd.3.smithi003.stdout:Jul 20 01:39:02 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-3[114976]: 2023-07-20T01:39:02.821+0000 7f56317f4700 -1 osd.3 0 waiting for initial osdmap 2023-07-20T01:39:03.190 INFO:journalctl@ceph.osd.3.smithi003.stdout:Jul 20 01:39:02 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-3[114976]: 2023-07-20T01:39:02.828+0000 7f562a5e4700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:39:03.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi043:vg_nvme/lv_3 2023-07-20T01:39:03.978 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:39:04.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:39:04.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T01:39:04.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:04.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:04.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:04.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:04.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:04.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:04.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:04.107 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:04.107 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:04.107 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:39:04.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T01:39:04.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:04.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:04.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:04.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:04.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:04.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:04.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:04.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:04.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:04.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.217 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:39:04.217 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T01:39:04.217 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:04.218 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:04.218 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.218 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.218 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:04.218 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:04.218 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.218 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:04.219 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:04.219 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:04.219 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:04.219 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:04.219 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:04.219 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:05.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:04 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:39:05.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:04 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:39:05.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:04 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi003 2023-07-20T01:39:05.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:04 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi003 to 8586M 2023-07-20T01:39:05.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:04 smithi086 ceph-mon[98138]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:39:05.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:04 smithi086 ceph-mon[98138]: osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193] boot 2023-07-20T01:39:05.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:04 smithi086 ceph-mon[98138]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T01:39:05.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:04 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:05.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:04 smithi086 ceph-mon[98138]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T01:39:05.167 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.198+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/2497994857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5bc100900 msgr2=0x7fd5bc100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:05.168 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.198+0000 7fd5c2bc7700 1 --2- 172.21.15.43:0/2497994857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5bc100900 0x7fd5bc100d60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80099c0 tx=0x7fd5a8009cd0 comp rx=0 tx=0).stop 2023-07-20T01:39:05.168 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.198+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/2497994857 shutdown_connections 2023-07-20T01:39:05.168 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.198+0000 7fd5c2bc7700 1 --2- 172.21.15.43:0/2497994857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5bc1012a0 0x7fd5bc107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:05.168 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.198+0000 7fd5c2bc7700 1 --2- 172.21.15.43:0/2497994857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5bc0ff700 0x7fd5bc0ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:05.168 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.198+0000 7fd5c2bc7700 1 --2- 172.21.15.43:0/2497994857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5bc100900 0x7fd5bc100d60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:05.169 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.198+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/2497994857 >> 172.21.15.43:0/2497994857 conn(0x7fd5bc0fac80 msgr2=0x7fd5bc0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:39:05.169 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.198+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/2497994857 shutdown_connections 2023-07-20T01:39:05.169 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.198+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/2497994857 wait complete. 2023-07-20T01:39:05.171 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c2bc7700 1 Processor -- start 2023-07-20T01:39:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c2bc7700 1 -- start start 2023-07-20T01:39:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c2bc7700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5bc0ff700 0x7fd5bc068df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c2bc7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5bc100900 0x7fd5bc19b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c2bc7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5bc1012a0 0x7fd5bc19d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c2bc7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc19dfb0 con 0x7fd5bc1012a0 2023-07-20T01:39:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c2bc7700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc19e120 con 0x7fd5bc0ff700 2023-07-20T01:39:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c2bc7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc19e2e0 con 0x7fd5bc100900 2023-07-20T01:39:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c0963700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5bc0ff700 0x7fd5bc068df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c0963700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5bc0ff700 0x7fd5bc068df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48206/0 (socket says 172.21.15.43:48206) 2023-07-20T01:39:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c0963700 1 -- 172.21.15.43:0/90598353 learned_addr learned my addr 172.21.15.43:0/90598353 (peer_addr_for_me v2:172.21.15.43:0/0) 2023-07-20T01:39:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5bbfff700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5bc100900 0x7fd5bc19b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c1164700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5bc1012a0 0x7fd5bc19d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c0963700 1 -- 172.21.15.43:0/90598353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5bc100900 msgr2=0x7fd5bc19b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c0963700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5bc100900 0x7fd5bc19b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c0963700 1 -- 172.21.15.43:0/90598353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5bc1012a0 msgr2=0x7fd5bc19d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c0963700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5bc1012a0 0x7fd5bc19d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.199+0000 7fd5c0963700 1 -- 172.21.15.43:0/90598353 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a80095f0 con 0x7fd5bc0ff700 2023-07-20T01:39:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.200+0000 7fd5c1164700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5bc1012a0 0x7fd5bc19d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:39:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.200+0000 7fd5c0963700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5bc0ff700 0x7fd5bc068df0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd5b000c970 tx=0x7fd5b000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:39:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.200+0000 7fd5b9ffb700 1 -- 172.21.15.43:0/90598353 <== mon.2 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0004220 con 0x7fd5bc0ff700 2023-07-20T01:39:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.200+0000 7fd5b9ffb700 1 -- 172.21.15.43:0/90598353 <== mon.2 v2:172.21.15.43:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5b0007d60 con 0x7fd5bc0ff700 2023-07-20T01:39:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.200+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/90598353 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5bc194fb0 con 0x7fd5bc0ff700 2023-07-20T01:39:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.200+0000 7fd5b9ffb700 1 -- 172.21.15.43:0/90598353 <== mon.2 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0003c90 con 0x7fd5bc0ff700 2023-07-20T01:39:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.200+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/90598353 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5bc195500 con 0x7fd5bc0ff700 2023-07-20T01:39:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.201+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/90598353 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5bc106d00 con 0x7fd5bc0ff700 2023-07-20T01:39:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.203+0000 7fd5b9ffb700 1 -- 172.21.15.43:0/90598353 <== mon.2 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fd5b0004380 con 0x7fd5bc0ff700 2023-07-20T01:39:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.203+0000 7fd5b9ffb700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd5ac0a11d0 0x7fd5ac0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.203+0000 7fd5b9ffb700 1 -- 172.21.15.43:0/90598353 <== mon.2 v2:172.21.15.43:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7fd5b00cb5f0 con 0x7fd5bc0ff700 2023-07-20T01:39:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.203+0000 7fd5bbfff700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd5ac0a11d0 0x7fd5ac0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.204+0000 7fd5bbfff700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd5ac0a11d0 0x7fd5ac0a3690 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800b580 tx=0x7fd5a8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:39:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.207+0000 7fd5b9ffb700 1 -- 172.21.15.43:0/90598353 <== mon.2 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd5b0095fc0 con 0x7fd5bc0ff700 2023-07-20T01:39:05.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:04 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:39:05.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:04 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:39:05.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:04 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi003 2023-07-20T01:39:05.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:04 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi003 to 8586M 2023-07-20T01:39:05.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:04 smithi003 ceph-mon[96422]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:39:05.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:04 smithi003 ceph-mon[96422]: osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193] boot 2023-07-20T01:39:05.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:04 smithi003 ceph-mon[96422]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T01:39:05.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:04 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:05.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:04 smithi003 ceph-mon[96422]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T01:39:05.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:04 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:39:05.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:04 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:39:05.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:04 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi003 2023-07-20T01:39:05.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:04 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi003 to 8586M 2023-07-20T01:39:05.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:04 smithi043 ceph-mon[97823]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:39:05.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:04 smithi043 ceph-mon[97823]: osd.3 [v2:172.21.15.3:6810/535233193,v1:172.21.15.3:6811/535233193] boot 2023-07-20T01:39:05.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:04 smithi043 ceph-mon[97823]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T01:39:05.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:04 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:39:05.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:04 smithi043 ceph-mon[97823]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T01:39:05.383 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:05.413+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/90598353 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5bc1969a0 con 0x7fd5ac0a11d0 2023-07-20T01:39:06.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:39:06.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:39:06.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:06.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:05 smithi086 ceph-mon[98138]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T01:39:06.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:39:06.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:39:06.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:06.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:05 smithi003 ceph-mon[96422]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T01:39:06.190 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:39:06.190 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:39:06.190 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:06.190 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:05 smithi043 ceph-mon[97823]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T01:39:07.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:06 smithi086 ceph-mon[98138]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:39:07.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:06 smithi086 ceph-mon[98138]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:07.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:06 smithi003 ceph-mon[96422]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:39:07.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:06 smithi003 ceph-mon[96422]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:07.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:06 smithi043 ceph-mon[97823]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:39:07.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:06 smithi043 ceph-mon[97823]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:08.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:07 smithi086 ceph-mon[98138]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T01:39:08.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:07 smithi086 ceph-mon[98138]: from='client.? 172.21.15.43:0/3963033318' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a35bbae-6f6c-4296-9459-f1b4d2de632f"}]: dispatch 2023-07-20T01:39:08.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:07 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a35bbae-6f6c-4296-9459-f1b4d2de632f"}]: dispatch 2023-07-20T01:39:08.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:07 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a35bbae-6f6c-4296-9459-f1b4d2de632f"}]': finished 2023-07-20T01:39:08.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:07 smithi086 ceph-mon[98138]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T01:39:08.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:08.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:07 smithi003 ceph-mon[96422]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T01:39:08.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:07 smithi003 ceph-mon[96422]: from='client.? 172.21.15.43:0/3963033318' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a35bbae-6f6c-4296-9459-f1b4d2de632f"}]: dispatch 2023-07-20T01:39:08.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:07 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a35bbae-6f6c-4296-9459-f1b4d2de632f"}]: dispatch 2023-07-20T01:39:08.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:07 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a35bbae-6f6c-4296-9459-f1b4d2de632f"}]': finished 2023-07-20T01:39:08.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:07 smithi003 ceph-mon[96422]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T01:39:08.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:08.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:07 smithi043 ceph-mon[97823]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T01:39:08.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:07 smithi043 ceph-mon[97823]: from='client.? 172.21.15.43:0/3963033318' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a35bbae-6f6c-4296-9459-f1b4d2de632f"}]: dispatch 2023-07-20T01:39:08.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:07 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a35bbae-6f6c-4296-9459-f1b4d2de632f"}]: dispatch 2023-07-20T01:39:08.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:07 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a35bbae-6f6c-4296-9459-f1b4d2de632f"}]': finished 2023-07-20T01:39:08.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:07 smithi043 ceph-mon[97823]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T01:39:08.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:09.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:08 smithi086 ceph-mon[98138]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 145 KiB/s, 0 objects/s recovering 2023-07-20T01:39:09.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:08 smithi086 ceph-mon[98138]: from='client.? 172.21.15.43:0/2523691888' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:39:09.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:08 smithi003 ceph-mon[96422]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 145 KiB/s, 0 objects/s recovering 2023-07-20T01:39:09.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:08 smithi003 ceph-mon[96422]: from='client.? 172.21.15.43:0/2523691888' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:39:09.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:08 smithi043 ceph-mon[97823]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 145 KiB/s, 0 objects/s recovering 2023-07-20T01:39:09.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:08 smithi043 ceph-mon[97823]: from='client.? 172.21.15.43:0/2523691888' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:39:11.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:10 smithi086 ceph-mon[98138]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 116 KiB/s, 0 objects/s recovering 2023-07-20T01:39:11.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:10 smithi003 ceph-mon[96422]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 116 KiB/s, 0 objects/s recovering 2023-07-20T01:39:11.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:10 smithi043 ceph-mon[97823]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 116 KiB/s, 0 objects/s recovering 2023-07-20T01:39:13.100 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:12 smithi043 ceph-mon[97823]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T01:39:13.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:12 smithi086 ceph-mon[98138]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T01:39:13.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:12 smithi003 ceph-mon[96422]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T01:39:15.064 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:14 smithi043 ceph-mon[97823]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:39:15.064 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:14 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T01:39:15.065 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:14 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:15.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:14 smithi086 ceph-mon[98138]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:39:15.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:14 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T01:39:15.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:14 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:15.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:14 smithi003 ceph-mon[96422]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:39:15.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:14 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T01:39:15.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:14 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:16.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:15 smithi086 ceph-mon[98138]: Deploying daemon osd.4 on smithi043 2023-07-20T01:39:16.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:15 smithi086 ceph-mon[98138]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:16.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:15 smithi003 ceph-mon[96422]: Deploying daemon osd.4 on smithi043 2023-07-20T01:39:16.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:15 smithi003 ceph-mon[96422]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:16.202 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:15 smithi043 ceph-mon[97823]: Deploying daemon osd.4 on smithi043 2023-07-20T01:39:16.202 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:15 smithi043 ceph-mon[97823]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:18.584 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:18 smithi043 ceph-mon[97823]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:18.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:18 smithi086 ceph-mon[98138]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:18.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:18 smithi003 ceph-mon[96422]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:20.132 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:19 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:20.132 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:19 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:20.132 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:19 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:20.132 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:19 smithi043 ceph-mon[97823]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:20.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:20.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:20.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:19 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:20.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:19 smithi003 ceph-mon[96422]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:20.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:20.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:20.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:19 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:20.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:19 smithi086 ceph-mon[98138]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:21.754 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 4 on host 'smithi043' 2023-07-20T01:39:21.755 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.784+0000 7fd5b9ffb700 1 -- 172.21.15.43:0/90598353 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd5bc1969a0 con 0x7fd5ac0a11d0 2023-07-20T01:39:21.756 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.786+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/90598353 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd5ac0a11d0 msgr2=0x7fd5ac0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:21.756 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.786+0000 7fd5c2bc7700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd5ac0a11d0 0x7fd5ac0a3690 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800b580 tx=0x7fd5a8000bc0 comp rx=0 tx=0).stop 2023-07-20T01:39:21.757 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.787+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/90598353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5bc0ff700 msgr2=0x7fd5bc068df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:21.757 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.787+0000 7fd5c2bc7700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5bc0ff700 0x7fd5bc068df0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd5b000c970 tx=0x7fd5b000cc80 comp rx=0 tx=0).stop 2023-07-20T01:39:21.757 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.787+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/90598353 shutdown_connections 2023-07-20T01:39:21.757 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.787+0000 7fd5c2bc7700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5bc1012a0 0x7fd5bc19d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:21.757 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.787+0000 7fd5c2bc7700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd5ac0a11d0 0x7fd5ac0a3690 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:21.757 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.787+0000 7fd5c2bc7700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5bc0ff700 0x7fd5bc068df0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:21.757 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.787+0000 7fd5c2bc7700 1 --2- 172.21.15.43:0/90598353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5bc100900 0x7fd5bc19b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:21.758 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.787+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/90598353 >> 172.21.15.43:0/90598353 conn(0x7fd5bc0fac80 msgr2=0x7fd5bc103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:39:21.758 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.787+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/90598353 shutdown_connections 2023-07-20T01:39:21.758 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:39:21.787+0000 7fd5c2bc7700 1 -- 172.21.15.43:0/90598353 wait complete. 2023-07-20T01:39:22.181 DEBUG:teuthology.orchestra.run.smithi043:osd.4> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.4.service 2023-07-20T01:39:22.183 INFO:tasks.cephadm:Deploying osd.5 on smithi086 with /dev/vg_nvme/lv_3... 2023-07-20T01:39:22.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T01:39:22.214 INFO:journalctl@ceph.osd.4.smithi043.stdout:-- Logs begin at Thu 2023-07-20 01:20:44 UTC. -- 2023-07-20T01:39:22.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:22.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:22.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:22 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:22.505 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:22 smithi043 ceph-mon[97823]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:22.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:22 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:22 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:22 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:22.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:22 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:22.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:22 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:22 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:22 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:22 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:22.606 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:22 smithi086 ceph-mon[98138]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:22.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:22 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:22 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:22 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:22.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:22 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:22.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:22 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:22 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:22 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:22.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:22 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:22.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:22 smithi003 ceph-mon[96422]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:22.743 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:39:22.980 INFO:journalctl@ceph.osd.4.smithi043.stdout:Jul 20 01:39:22 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-4[108497]: 2023-07-20T01:39:22.899+0000 7f9c02975640 -1 osd.4 0 log_to_monitors true 2023-07-20T01:39:24.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:23 smithi043 ceph-mon[97823]: from='osd.4 [v2:172.21.15.43:6808/47431618,v1:172.21.15.43:6809/47431618]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:39:24.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:23 smithi043 ceph-mon[97823]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:39:24.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:23 smithi086 ceph-mon[98138]: from='osd.4 [v2:172.21.15.43:6808/47431618,v1:172.21.15.43:6809/47431618]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:39:24.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:23 smithi086 ceph-mon[98138]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:39:24.110 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T01:39:24.110 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T01:39:24.111 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2023-07-20T01:39:24.111 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2023-07-20T01:39:24.111 INFO:teuthology.orchestra.run.smithi086.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110839 s, 946 MB/s 2023-07-20T01:39:24.111 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2023-07-20T01:39:24.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:23 smithi003 ceph-mon[96422]: from='osd.4 [v2:172.21.15.43:6808/47431618,v1:172.21.15.43:6809/47431618]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:39:24.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:23 smithi003 ceph-mon[96422]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:39:24.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi086:vg_nvme/lv_3 2023-07-20T01:39:25.003 INFO:journalctl@ceph.osd.4.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-4[108497]: 2023-07-20T01:39:24.762+0000 7f9bf77ec700 -1 osd.4 0 waiting for initial osdmap 2023-07-20T01:39:25.004 INFO:journalctl@ceph.osd.4.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-4[108497]: 2023-07-20T01:39:24.769+0000 7f9bf1ddf700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:39:25.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T01:39:25.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T01:39:25.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:25.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='osd.4 [v2:172.21.15.43:6808/47431618,v1:172.21.15.43:6809/47431618]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:39:25.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:39:25.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:25.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:25.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:25.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.006 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:25.006 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:25.006 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:24 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.050 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:39:25.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T01:39:25.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T01:39:25.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:25.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='osd.4 [v2:172.21.15.43:6808/47431618,v1:172.21.15.43:6809/47431618]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:39:25.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:39:25.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:25.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:25.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:25.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:25.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:25.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:24 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T01:39:25.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T01:39:25.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:25.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='osd.4 [v2:172.21.15.43:6808/47431618,v1:172.21.15.43:6809/47431618]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:39:25.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:39:25.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:25.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:25.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:25.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:25.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:25.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:24 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:25.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:25 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi043 2023-07-20T01:39:25.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:25 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi043 to 8586M 2023-07-20T01:39:25.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:25 smithi086 ceph-mon[98138]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:39:25.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:25 smithi086 ceph-mon[98138]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T01:39:25.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:25 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:25.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:25 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:26.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:25 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi043 2023-07-20T01:39:26.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:25 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi043 to 8586M 2023-07-20T01:39:26.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:25 smithi043 ceph-mon[97823]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:39:26.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:25 smithi043 ceph-mon[97823]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T01:39:26.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:25 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:26.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:25 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:26.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:25 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi043 2023-07-20T01:39:26.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:25 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi043 to 8586M 2023-07-20T01:39:26.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:25 smithi003 ceph-mon[96422]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:39:26.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:25 smithi003 ceph-mon[96422]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T01:39:26.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:25 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:26.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:25 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:26.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.356+0000 7f75f66e6700 1 -- 172.21.15.86:0/1290168567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f0100310 msgr2=0x7f75f0107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:26.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.356+0000 7f75f66e6700 1 --2- 172.21.15.86:0/1290168567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f0100310 0x7f75f0107bb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f75e800b0a0 tx=0x7f75e800b3b0 comp rx=0 tx=0).stop 2023-07-20T01:39:26.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.356+0000 7f75f66e6700 1 -- 172.21.15.86:0/1290168567 shutdown_connections 2023-07-20T01:39:26.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.356+0000 7f75f66e6700 1 --2- 172.21.15.86:0/1290168567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f0100310 0x7f75f0107bb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:26.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.356+0000 7f75f66e6700 1 --2- 172.21.15.86:0/1290168567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75f0100d80 0x7f75f01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:26.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.356+0000 7f75f66e6700 1 --2- 172.21.15.86:0/1290168567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75f00ff950 0x7f75f00ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:26.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.356+0000 7f75f66e6700 1 -- 172.21.15.86:0/1290168567 >> 172.21.15.86:0/1290168567 conn(0x7f75f00fa420 msgr2=0x7f75f00fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:39:26.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.356+0000 7f75f66e6700 1 -- 172.21.15.86:0/1290168567 shutdown_connections 2023-07-20T01:39:26.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.356+0000 7f75f66e6700 1 -- 172.21.15.86:0/1290168567 wait complete. 2023-07-20T01:39:26.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.356+0000 7f75f66e6700 1 Processor -- start 2023-07-20T01:39:26.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.356+0000 7f75f66e6700 1 -- start start 2023-07-20T01:39:26.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75f66e6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f00ff950 0x7f75f00ff3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:26.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75f66e6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75f0100310 0x7f75f019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:26.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75f66e6700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75f0100d80 0x7f75f019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:26.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75f66e6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f01a0250 con 0x7f75f0100310 2023-07-20T01:39:26.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75f66e6700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f01a03c0 con 0x7f75f0100d80 2023-07-20T01:39:26.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75f66e6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f01a0580 con 0x7f75f00ff950 2023-07-20T01:39:26.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75effff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f00ff950 0x7f75f00ff3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:26.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75effff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f00ff950 0x7f75f00ff3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58520/0 (socket says 172.21.15.86:58520) 2023-07-20T01:39:26.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75effff700 1 -- 172.21.15.86:0/376807796 learned_addr learned my addr 172.21.15.86:0/376807796 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-07-20T01:39:26.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75ef7fe700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75f0100310 0x7f75f019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:26.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75effff700 1 -- 172.21.15.86:0/376807796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75f0100d80 msgr2=0x7f75f019fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:26.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75f4c83700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75f0100d80 0x7f75f019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:26.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75effff700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75f0100d80 0x7f75f019fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:26.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75effff700 1 -- 172.21.15.86:0/376807796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75f0100310 msgr2=0x7f75f019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:26.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75effff700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75f0100310 0x7f75f019d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:26.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75effff700 1 -- 172.21.15.86:0/376807796 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e8009d00 con 0x7f75f00ff950 2023-07-20T01:39:26.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75effff700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f00ff950 0x7f75f00ff3a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f75d800c920 tx=0x7f75d800cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:39:26.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75ed7fa700 1 -- 172.21.15.86:0/376807796 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d8007b00 con 0x7f75f00ff950 2023-07-20T01:39:26.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75ed7fa700 1 -- 172.21.15.86:0/376807796 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75d8004cf0 con 0x7f75f00ff950 2023-07-20T01:39:26.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75f66e6700 1 -- 172.21.15.86:0/376807796 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f01a0860 con 0x7f75f00ff950 2023-07-20T01:39:26.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.357+0000 7f75ed7fa700 1 -- 172.21.15.86:0/376807796 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d8005610 con 0x7f75f00ff950 2023-07-20T01:39:26.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.358+0000 7f75f66e6700 1 -- 172.21.15.86:0/376807796 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f01a0d60 con 0x7f75f00ff950 2023-07-20T01:39:26.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.358+0000 7f75f66e6700 1 -- 172.21.15.86:0/376807796 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f010af10 con 0x7f75f00ff950 2023-07-20T01:39:26.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.360+0000 7f75ed7fa700 1 -- 172.21.15.86:0/376807796 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f75d8004750 con 0x7f75f00ff950 2023-07-20T01:39:26.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.360+0000 7f75ed7fa700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f75dc0a1150 0x7f75dc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:26.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.360+0000 7f75ed7fa700 1 -- 172.21.15.86:0/376807796 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f75d80cb8b0 con 0x7f75f00ff950 2023-07-20T01:39:26.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.360+0000 7f75ef7fe700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f75dc0a1150 0x7f75dc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:26.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.361+0000 7f75ef7fe700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f75dc0a1150 0x7f75dc0a3610 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75f00ff920 tx=0x7f75e0005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:39:26.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.364+0000 7f75ed7fa700 1 -- 172.21.15.86:0/376807796 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f75d8096070 con 0x7f75f00ff950 2023-07-20T01:39:26.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:26.555+0000 7f75f66e6700 1 -- 172.21.15.86:0/376807796 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f75f0060bb0 con 0x7f75dc0a1150 2023-07-20T01:39:27.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:26 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:39:27.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:26 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:39:27.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:26 smithi043 ceph-mon[97823]: osd.4 [v2:172.21.15.43:6808/47431618,v1:172.21.15.43:6809/47431618] boot 2023-07-20T01:39:27.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:26 smithi043 ceph-mon[97823]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T01:39:27.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:27.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:26 smithi043 ceph-mon[97823]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:27.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:39:27.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:39:27.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:27.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:26 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:39:27.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:26 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:39:27.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:26 smithi086 ceph-mon[98138]: osd.4 [v2:172.21.15.43:6808/47431618,v1:172.21.15.43:6809/47431618] boot 2023-07-20T01:39:27.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:26 smithi086 ceph-mon[98138]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T01:39:27.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:27.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:26 smithi086 ceph-mon[98138]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:27.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:39:27.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:39:27.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:27.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:26 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:39:27.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:26 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:39:27.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:26 smithi003 ceph-mon[96422]: osd.4 [v2:172.21.15.43:6808/47431618,v1:172.21.15.43:6809/47431618] boot 2023-07-20T01:39:27.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:26 smithi003 ceph-mon[96422]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T01:39:27.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:39:27.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:26 smithi003 ceph-mon[96422]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:39:27.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:39:27.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:39:27.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:27.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:27 smithi086 ceph-mon[98138]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:39:27.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:27 smithi086 ceph-mon[98138]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T01:39:28.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:27 smithi043 ceph-mon[97823]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:39:28.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:27 smithi043 ceph-mon[97823]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T01:39:28.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:27 smithi003 ceph-mon[96422]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:39:28.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:27 smithi003 ceph-mon[96422]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T01:39:29.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:28 smithi043 ceph-mon[97823]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:29.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:28 smithi086 ceph-mon[98138]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:29.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:28 smithi003 ceph-mon[96422]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:30.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:29 smithi086 ceph-mon[98138]: from='client.? 172.21.15.86:0/1011844381' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34cccdff-563b-413c-b56c-08839b1534aa"}]: dispatch 2023-07-20T01:39:30.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:29 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34cccdff-563b-413c-b56c-08839b1534aa"}]: dispatch 2023-07-20T01:39:30.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:29 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34cccdff-563b-413c-b56c-08839b1534aa"}]': finished 2023-07-20T01:39:30.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:29 smithi086 ceph-mon[98138]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T01:39:30.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:30.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:29 smithi086 ceph-mon[98138]: from='client.? 172.21.15.86:0/1650705725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:39:30.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:29 smithi003 ceph-mon[96422]: from='client.? 172.21.15.86:0/1011844381' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34cccdff-563b-413c-b56c-08839b1534aa"}]: dispatch 2023-07-20T01:39:30.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:29 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34cccdff-563b-413c-b56c-08839b1534aa"}]: dispatch 2023-07-20T01:39:30.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:29 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34cccdff-563b-413c-b56c-08839b1534aa"}]': finished 2023-07-20T01:39:30.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:29 smithi003 ceph-mon[96422]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T01:39:30.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:30.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:29 smithi003 ceph-mon[96422]: from='client.? 172.21.15.86:0/1650705725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:39:30.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:29 smithi043 ceph-mon[97823]: from='client.? 172.21.15.86:0/1011844381' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34cccdff-563b-413c-b56c-08839b1534aa"}]: dispatch 2023-07-20T01:39:30.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:29 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34cccdff-563b-413c-b56c-08839b1534aa"}]: dispatch 2023-07-20T01:39:30.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:29 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34cccdff-563b-413c-b56c-08839b1534aa"}]': finished 2023-07-20T01:39:30.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:29 smithi043 ceph-mon[97823]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T01:39:30.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:30.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:29 smithi043 ceph-mon[97823]: from='client.? 172.21.15.86:0/1650705725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:39:31.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:30 smithi086 ceph-mon[98138]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:31.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:30 smithi003 ceph-mon[96422]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:31.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:30 smithi043 ceph-mon[97823]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:33.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:32 smithi086 ceph-mon[98138]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:33.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:32 smithi003 ceph-mon[96422]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:33.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:32 smithi043 ceph-mon[97823]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:35.071 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:34 smithi086 ceph-mon[98138]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:35.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:34 smithi003 ceph-mon[96422]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:35.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:34 smithi043 ceph-mon[97823]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:36.095 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T01:39:36.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:35 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:36.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T01:39:36.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:35 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:36.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T01:39:36.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:35 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:37.042 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:36 smithi086 ceph-mon[98138]: Deploying daemon osd.5 on smithi086 2023-07-20T01:39:37.043 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:36 smithi086 ceph-mon[98138]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:37.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:36 smithi003 ceph-mon[96422]: Deploying daemon osd.5 on smithi086 2023-07-20T01:39:37.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:36 smithi003 ceph-mon[96422]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:37.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:36 smithi043 ceph-mon[97823]: Deploying daemon osd.5 on smithi086 2023-07-20T01:39:37.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:36 smithi043 ceph-mon[97823]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:38.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:38 smithi086 ceph-mon[98138]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:38.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:38 smithi003 ceph-mon[96422]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:38.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:38 smithi043 ceph-mon[97823]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:40.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:40 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:40.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:40 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:40.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:40 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:40.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:40 smithi086 ceph-mon[98138]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:40.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:40 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:40.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:40 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:40.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:40 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:40.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:40 smithi043 ceph-mon[97823]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:40.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:40 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:40.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:40 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:40.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:40 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:40.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:40 smithi003 ceph-mon[96422]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:42.033 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.064+0000 7f75ed7fa700 1 -- 172.21.15.86:0/376807796 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f75f0060bb0 con 0x7f75dc0a1150 2023-07-20T01:39:42.034 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 5 on host 'smithi086' 2023-07-20T01:39:42.035 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 -- 172.21.15.86:0/376807796 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f75dc0a1150 msgr2=0x7f75dc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:42.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f75dc0a1150 0x7f75dc0a3610 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75f00ff920 tx=0x7f75e0005d50 comp rx=0 tx=0).stop 2023-07-20T01:39:42.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 -- 172.21.15.86:0/376807796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f00ff950 msgr2=0x7f75f00ff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:42.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f00ff950 0x7f75f00ff3a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f75d800c920 tx=0x7f75d800cce0 comp rx=0 tx=0).stop 2023-07-20T01:39:42.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 -- 172.21.15.86:0/376807796 shutdown_connections 2023-07-20T01:39:42.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f75dc0a1150 0x7f75dc0a3610 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:42.036 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75f0100d80 0x7f75f019fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:42.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f00ff950 0x7f75f00ff3a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:42.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 --2- 172.21.15.86:0/376807796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75f0100310 0x7f75f019d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:42.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 -- 172.21.15.86:0/376807796 >> 172.21.15.86:0/376807796 conn(0x7f75f00fa420 msgr2=0x7f75f00fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:39:42.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 -- 172.21.15.86:0/376807796 shutdown_connections 2023-07-20T01:39:42.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:39:42.067+0000 7f75f66e6700 1 -- 172.21.15.86:0/376807796 wait complete. 2023-07-20T01:39:42.509 DEBUG:teuthology.orchestra.run.smithi086:osd.5> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.5.service 2023-07-20T01:39:42.512 INFO:tasks.cephadm:Deploying osd.6 on smithi003 with /dev/vg_nvme/lv_2... 2023-07-20T01:39:42.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T01:39:42.543 INFO:journalctl@ceph.osd.5.smithi086.stdout:-- Logs begin at Thu 2023-07-20 01:20:49 UTC. -- 2023-07-20T01:39:42.810 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:42 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:42.810 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:42 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:42.810 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:42 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:42.810 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:42 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:42.810 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:42 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:42.811 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:42 smithi086 ceph-mon[98138]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:42.811 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:42 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:42.811 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:42 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:42.811 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:42 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:42.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:42 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:42.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:42 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:42.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:42 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:42.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:42 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:42.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:42 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:42.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:42 smithi003 ceph-mon[96422]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:42.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:42 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:42.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:42 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:42.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:42 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:43.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:42 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:43.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:42 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:43.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:42 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:43.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:42 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:42 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:42 smithi043 ceph-mon[97823]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:42 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:39:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:42 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:43.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:42 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:43.087 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:39:43.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:43 smithi086 ceph-mon[98138]: from='osd.5 [v2:172.21.15.86:6808/1573392100,v1:172.21.15.86:6809/1573392100]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:39:43.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:43 smithi086 ceph-mon[98138]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:39:43.855 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:39:43 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:39:43.449+0000 7f8a7cee5640 -1 osd.5 0 log_to_monitors true 2023-07-20T01:39:43.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:43 smithi003 ceph-mon[96422]: from='osd.5 [v2:172.21.15.86:6808/1573392100,v1:172.21.15.86:6809/1573392100]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:39:43.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:43 smithi003 ceph-mon[96422]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:39:44.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:43 smithi043 ceph-mon[97823]: from='osd.5 [v2:172.21.15.86:6808/1573392100,v1:172.21.15.86:6809/1573392100]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:39:44.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:43 smithi043 ceph-mon[97823]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:39:44.505 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T01:39:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T01:39:44.506 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2023-07-20T01:39:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2023-07-20T01:39:44.507 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0106907 s, 981 MB/s 2023-07-20T01:39:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2023-07-20T01:39:44.853 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:39:44.585+0000 7f8a71d5c700 -1 osd.5 0 waiting for initial osdmap 2023-07-20T01:39:44.854 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:39:44.591+0000 7f8a6b34d700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:39:44.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T01:39:44.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T01:39:44.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:44.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='osd.5 [v2:172.21.15.86:6808/1573392100,v1:172.21.15.86:6809/1573392100]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:39:44.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:39:44.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:44.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:44.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:44.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:44.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:44.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:44.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:44.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:44.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:44 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:44.908 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T01:39:44.909 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T01:39:44.909 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:44.909 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='osd.5 [v2:172.21.15.86:6808/1573392100,v1:172.21.15.86:6809/1573392100]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:39:44.909 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:39:44.909 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:44.909 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:44.909 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:44.910 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:44.910 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:44.910 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:44.910 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:44.910 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:44.910 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:44 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:44.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi003:vg_nvme/lv_2 2023-07-20T01:39:45.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T01:39:45.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T01:39:45.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:45.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='osd.5 [v2:172.21.15.86:6808/1573392100,v1:172.21.15.86:6809/1573392100]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:39:45.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:39:45.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:45.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:45.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:45.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:45.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:39:45.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:45.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:45.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:39:45.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:44 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:39:45.501 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:39:45.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:45 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi086 2023-07-20T01:39:45.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:45 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi086 to 8584M 2023-07-20T01:39:45.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:45 smithi086 ceph-mon[98138]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:39:45.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:45 smithi086 ceph-mon[98138]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T01:39:45.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:45 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:45.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:45 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:45.870 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:45 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi086 2023-07-20T01:39:45.870 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:45 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi086 to 8584M 2023-07-20T01:39:45.870 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:45 smithi003 ceph-mon[96422]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:39:45.870 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:45 smithi003 ceph-mon[96422]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T01:39:45.871 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:45 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:45.871 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:45 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:46.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:45 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi086 2023-07-20T01:39:46.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:45 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi086 to 8584M 2023-07-20T01:39:46.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:45 smithi043 ceph-mon[97823]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:39:46.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:45 smithi043 ceph-mon[97823]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T01:39:46.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:45 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:46.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:45 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:46.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.755+0000 7ff5f419b700 1 -- 172.21.15.3:0/2804172633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec100310 msgr2=0x7ff5ec107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:46.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.755+0000 7ff5f419b700 1 --2- 172.21.15.3:0/2804172633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec100310 0x7ff5ec107bb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400b0a0 tx=0x7ff5e400b3b0 comp rx=0 tx=0).stop 2023-07-20T01:39:46.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.755+0000 7ff5f419b700 1 -- 172.21.15.3:0/2804172633 shutdown_connections 2023-07-20T01:39:46.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.755+0000 7ff5f419b700 1 --2- 172.21.15.3:0/2804172633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec100310 0x7ff5ec107bb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:46.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.755+0000 7ff5f419b700 1 --2- 172.21.15.3:0/2804172633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5ec100d80 0x7ff5ec1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:46.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.755+0000 7ff5f419b700 1 --2- 172.21.15.3:0/2804172633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5ec0ff950 0x7ff5ec0ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:46.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.755+0000 7ff5f419b700 1 -- 172.21.15.3:0/2804172633 >> 172.21.15.3:0/2804172633 conn(0x7ff5ec0fa420 msgr2=0x7ff5ec0fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:39:46.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.755+0000 7ff5f419b700 1 -- 172.21.15.3:0/2804172633 shutdown_connections 2023-07-20T01:39:46.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.755+0000 7ff5f419b700 1 -- 172.21.15.3:0/2804172633 wait complete. 2023-07-20T01:39:46.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f419b700 1 Processor -- start 2023-07-20T01:39:46.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f419b700 1 -- start start 2023-07-20T01:39:46.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f419b700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5ec0ff950 0x7ff5ec195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:46.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f419b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec100310 0x7ff5ec1959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:46.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f419b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5ec100d80 0x7ff5ec19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:46.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f419b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec19d020 con 0x7ff5ec100310 2023-07-20T01:39:46.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f419b700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec19d160 con 0x7ff5ec0ff950 2023-07-20T01:39:46.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f419b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec19d2f0 con 0x7ff5ec100d80 2023-07-20T01:39:46.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f1736700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec100310 0x7ff5ec1959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:46.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f1736700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec100310 0x7ff5ec1959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42084/0 (socket says 172.21.15.3:42084) 2023-07-20T01:39:46.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f1736700 1 -- 172.21.15.3:0/3348404639 learned_addr learned my addr 172.21.15.3:0/3348404639 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:39:46.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f1f37700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5ec0ff950 0x7ff5ec195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:46.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.756+0000 7ff5f2738700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5ec100d80 0x7ff5ec19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:46.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.757+0000 7ff5f1736700 1 -- 172.21.15.3:0/3348404639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5ec100d80 msgr2=0x7ff5ec19ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:46.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.757+0000 7ff5f1736700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5ec100d80 0x7ff5ec19ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:46.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.757+0000 7ff5f1736700 1 -- 172.21.15.3:0/3348404639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5ec0ff950 msgr2=0x7ff5ec195490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:39:46.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.757+0000 7ff5f1736700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5ec0ff950 0x7ff5ec195490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:39:46.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.757+0000 7ff5f1736700 1 -- 172.21.15.3:0/3348404639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e4009d00 con 0x7ff5ec100310 2023-07-20T01:39:46.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.757+0000 7ff5f1736700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec100310 0x7ff5ec1959d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00d9b0 tx=0x7ff5dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:39:46.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.757+0000 7ff5e2ffd700 1 -- 172.21.15.3:0/3348404639 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc004220 con 0x7ff5ec100310 2023-07-20T01:39:46.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.757+0000 7ff5f419b700 1 -- 172.21.15.3:0/3348404639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ec19d5d0 con 0x7ff5ec100310 2023-07-20T01:39:46.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.757+0000 7ff5e2ffd700 1 -- 172.21.15.3:0/3348404639 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5dc007640 con 0x7ff5ec100310 2023-07-20T01:39:46.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.757+0000 7ff5e2ffd700 1 -- 172.21.15.3:0/3348404639 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc005650 con 0x7ff5ec100310 2023-07-20T01:39:46.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.757+0000 7ff5f419b700 1 -- 172.21.15.3:0/3348404639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ec19dad0 con 0x7ff5ec100310 2023-07-20T01:39:46.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.758+0000 7ff5f419b700 1 -- 172.21.15.3:0/3348404639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ec106ce0 con 0x7ff5ec100310 2023-07-20T01:39:46.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.760+0000 7ff5e2ffd700 1 -- 172.21.15.3:0/3348404639 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7ff5dc004750 con 0x7ff5ec100310 2023-07-20T01:39:46.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.760+0000 7ff5e2ffd700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff5d80a1150 0x7ff5d80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:39:46.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.760+0000 7ff5f1f37700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff5d80a1150 0x7ff5d80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:39:46.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.760+0000 7ff5e2ffd700 1 -- 172.21.15.3:0/3348404639 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7ff5dc0cbe70 con 0x7ff5ec100310 2023-07-20T01:39:46.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.761+0000 7ff5f1f37700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff5d80a1150 0x7ff5d80a3610 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8007b60 tx=0x7ff5e80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:39:46.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.764+0000 7ff5e2ffd700 1 -- 172.21.15.3:0/3348404639 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff5dc096460 con 0x7ff5ec100310 2023-07-20T01:39:46.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:46 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:39:46.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:46 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:39:46.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:46 smithi086 ceph-mon[98138]: osd.5 [v2:172.21.15.86:6808/1573392100,v1:172.21.15.86:6809/1573392100] boot 2023-07-20T01:39:46.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:46 smithi086 ceph-mon[98138]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T01:39:46.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:46 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:46.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:46 smithi086 ceph-mon[98138]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:46.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:46 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:39:46.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:46 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:39:46.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:46 smithi003 ceph-mon[96422]: osd.5 [v2:172.21.15.86:6808/1573392100,v1:172.21.15.86:6809/1573392100] boot 2023-07-20T01:39:46.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:46 smithi003 ceph-mon[96422]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T01:39:46.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:46 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:46.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:46 smithi003 ceph-mon[96422]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:39:46.972+0000 7ff5f419b700 1 -- 172.21.15.3:0/3348404639 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5ec060bb0 con 0x7ff5d80a1150 2023-07-20T01:39:47.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:46 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:39:47.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:46 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:39:47.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:46 smithi043 ceph-mon[97823]: osd.5 [v2:172.21.15.86:6808/1573392100,v1:172.21.15.86:6809/1573392100] boot 2023-07-20T01:39:47.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:46 smithi043 ceph-mon[97823]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T01:39:47.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:46 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:39:47.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:46 smithi043 ceph-mon[97823]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:39:47.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:47 smithi086 ceph-mon[98138]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T01:39:47.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:47 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:39:47.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:47 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:39:47.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:47 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:47.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:47 smithi003 ceph-mon[96422]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T01:39:47.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:39:47.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:39:47.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:48.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:47 smithi043 ceph-mon[97823]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T01:39:48.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:47 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:39:48.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:47 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:39:48.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:47 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:48.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:48 smithi003 ceph-mon[96422]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:39:48.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:48 smithi003 ceph-mon[96422]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T01:39:48.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:48 smithi003 ceph-mon[96422]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:39:48.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:48 smithi086 ceph-mon[98138]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:39:48.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:48 smithi086 ceph-mon[98138]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T01:39:48.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:48 smithi086 ceph-mon[98138]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:39:49.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:48 smithi043 ceph-mon[97823]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:39:49.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:48 smithi043 ceph-mon[97823]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T01:39:49.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:48 smithi043 ceph-mon[97823]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:39:49.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:49 smithi086 ceph-mon[98138]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T01:39:49.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:49 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/637606044' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1c8f2b2-aed6-4277-b645-6a68e287ca2f"}]: dispatch 2023-07-20T01:39:49.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:49 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/637606044' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1c8f2b2-aed6-4277-b645-6a68e287ca2f"}]': finished 2023-07-20T01:39:49.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:49 smithi086 ceph-mon[98138]: osdmap e41: 7 total, 6 up, 7 in 2023-07-20T01:39:49.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:49 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:39:49.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:49 smithi003 ceph-mon[96422]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T01:39:49.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:49 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/637606044' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1c8f2b2-aed6-4277-b645-6a68e287ca2f"}]: dispatch 2023-07-20T01:39:49.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:49 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/637606044' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1c8f2b2-aed6-4277-b645-6a68e287ca2f"}]': finished 2023-07-20T01:39:49.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:49 smithi003 ceph-mon[96422]: osdmap e41: 7 total, 6 up, 7 in 2023-07-20T01:39:49.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:49 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:39:50.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:49 smithi043 ceph-mon[97823]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T01:39:50.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:49 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/637606044' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1c8f2b2-aed6-4277-b645-6a68e287ca2f"}]: dispatch 2023-07-20T01:39:50.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:49 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/637606044' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1c8f2b2-aed6-4277-b645-6a68e287ca2f"}]': finished 2023-07-20T01:39:50.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:49 smithi043 ceph-mon[97823]: osdmap e41: 7 total, 6 up, 7 in 2023-07-20T01:39:50.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:49 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:39:50.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:50 smithi086 ceph-mon[98138]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:39:50.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:50 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/4254797343' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:39:50.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:50 smithi003 ceph-mon[96422]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:39:50.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:50 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/4254797343' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:39:51.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:50 smithi043 ceph-mon[97823]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:39:51.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:50 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/4254797343' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:39:52.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:51 smithi003 ceph-mon[96422]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-20T01:39:52.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:51 smithi043 ceph-mon[97823]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-20T01:39:52.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:51 smithi086 ceph-mon[98138]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-20T01:39:54.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:53 smithi003 ceph-mon[96422]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-20T01:39:54.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:53 smithi043 ceph-mon[97823]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-20T01:39:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:53 smithi086 ceph-mon[98138]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-20T01:39:55.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:55 smithi003 ceph-mon[96422]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:39:56.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:55 smithi043 ceph-mon[97823]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:39:56.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:55 smithi086 ceph-mon[98138]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:39:57.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:56 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T01:39:57.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:56 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:57.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:56 smithi003 ceph-mon[96422]: Deploying daemon osd.6 on smithi003 2023-07-20T01:39:57.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:56 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T01:39:57.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:56 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:57.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:56 smithi043 ceph-mon[97823]: Deploying daemon osd.6 on smithi003 2023-07-20T01:39:57.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:56 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T01:39:57.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:56 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:39:57.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:56 smithi086 ceph-mon[98138]: Deploying daemon osd.6 on smithi003 2023-07-20T01:39:58.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:58 smithi086 ceph-mon[98138]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T01:39:58.612 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:58 smithi003 ceph-mon[96422]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T01:39:58.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:58 smithi043 ceph-mon[97823]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T01:40:00.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:39:59 smithi003 ceph-mon[96422]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-20T01:40:00.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:39:59 smithi043 ceph-mon[97823]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-20T01:40:00.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:39:59 smithi086 ceph-mon[98138]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-20T01:40:01.179 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:00 smithi003 ceph-mon[96422]: overall HEALTH_OK 2023-07-20T01:40:01.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:00 smithi043 ceph-mon[97823]: overall HEALTH_OK 2023-07-20T01:40:01.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:00 smithi086 ceph-mon[98138]: overall HEALTH_OK 2023-07-20T01:40:02.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:02 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:02.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:02 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:02.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:02 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:02.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:02 smithi003 ceph-mon[96422]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:40:02.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:02 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:02.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:02 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:02.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:02 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:02.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:02 smithi043 ceph-mon[97823]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:40:02.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:02 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:02.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:02 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:02.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:02 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:02.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:02 smithi086 ceph-mon[98138]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:40:04.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.333+0000 7ff5e2ffd700 1 -- 172.21.15.3:0/3348404639 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff5ec060bb0 con 0x7ff5d80a1150 2023-07-20T01:40:04.303 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 6 on host 'smithi003' 2023-07-20T01:40:04.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 -- 172.21.15.3:0/3348404639 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff5d80a1150 msgr2=0x7ff5d80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:04.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff5d80a1150 0x7ff5d80a3610 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8007b60 tx=0x7ff5e80073d0 comp rx=0 tx=0).stop 2023-07-20T01:40:04.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 -- 172.21.15.3:0/3348404639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec100310 msgr2=0x7ff5ec1959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:04.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec100310 0x7ff5ec1959d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00d9b0 tx=0x7ff5dc00dcc0 comp rx=0 tx=0).stop 2023-07-20T01:40:04.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 -- 172.21.15.3:0/3348404639 shutdown_connections 2023-07-20T01:40:04.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5ec100d80 0x7ff5ec19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:04.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff5d80a1150 0x7ff5d80a3610 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:04.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5ec0ff950 0x7ff5ec195490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:04.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 --2- 172.21.15.3:0/3348404639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec100310 0x7ff5ec1959d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:04.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 -- 172.21.15.3:0/3348404639 >> 172.21.15.3:0/3348404639 conn(0x7ff5ec0fa420 msgr2=0x7ff5ec103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:40:04.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 -- 172.21.15.3:0/3348404639 shutdown_connections 2023-07-20T01:40:04.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:04.336+0000 7ff5f419b700 1 -- 172.21.15.3:0/3348404639 wait complete. 2023-07-20T01:40:04.686 DEBUG:teuthology.orchestra.run.smithi003:osd.6> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.6.service 2023-07-20T01:40:04.688 INFO:tasks.cephadm:Deploying osd.7 on smithi043 with /dev/vg_nvme/lv_2... 2023-07-20T01:40:04.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T01:40:04.731 INFO:journalctl@ceph.osd.6.smithi003.stdout:-- Logs begin at Thu 2023-07-20 01:20:47 UTC. -- 2023-07-20T01:40:04.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:04 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:04.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:04 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:04.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:04 smithi003 ceph-mon[96422]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:40:04.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:04 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:04.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:04 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:04.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:04 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:04.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:04 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:04.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:04 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:04.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:04 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:05.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:04 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:05.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:04 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:05.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:04 smithi043 ceph-mon[97823]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:40:05.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:04 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:05.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:04 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:05.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:04 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:05.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:04 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:05.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:04 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:05.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:04 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:05.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:04 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:05.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:04 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:05.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:04 smithi086 ceph-mon[98138]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:40:05.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:04 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:05.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:04 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:05.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:04 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:05.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:04 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:05.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:04 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:05.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:04 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:05.273 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:40:05.546 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:40:05 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:40:05.234+0000 7fca22265640 -1 osd.6 0 log_to_monitors true 2023-07-20T01:40:05.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:05 smithi043 ceph-mon[97823]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T01:40:05.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:05 smithi003 ceph-mon[96422]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T01:40:06.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:05 smithi086 ceph-mon[98138]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T01:40:06.759 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T01:40:06.760 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T01:40:06.760 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2023-07-20T01:40:06.761 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2023-07-20T01:40:06.761 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0133033 s, 788 MB/s 2023-07-20T01:40:06.761 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2023-07-20T01:40:06.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:06 smithi003 ceph-mon[96422]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T01:40:06.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:06 smithi003 ceph-mon[96422]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T01:40:06.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:06 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:06.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:06 smithi003 ceph-mon[96422]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:40:06.788 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:06 smithi003 ceph-mon[96422]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:40:06.788 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:40:06 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:40:06.672+0000 7fca170dc700 -1 osd.6 0 waiting for initial osdmap 2023-07-20T01:40:06.788 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:40:06 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:40:06.683+0000 7fca0fecc700 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:40:06.899 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:06 smithi043 ceph-mon[97823]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T01:40:06.899 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:06 smithi043 ceph-mon[97823]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T01:40:06.899 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:06 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:06.899 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:06 smithi043 ceph-mon[97823]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:40:06.900 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:06 smithi043 ceph-mon[97823]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:40:07.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:06 smithi086 ceph-mon[98138]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T01:40:07.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:06 smithi086 ceph-mon[98138]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T01:40:07.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:06 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:07.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:06 smithi086 ceph-mon[98138]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:40:07.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:06 smithi086 ceph-mon[98138]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:40:07.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi043:vg_nvme/lv_2 2023-07-20T01:40:07.782 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:40:07.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:40:07.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T01:40:07.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:07.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:07.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:07.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:07.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:07.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:07.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:07.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:07.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:07.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:07.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:07.997 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:40:07.997 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T01:40:07.997 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:07.997 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:07.998 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:07.998 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:07.998 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:07.998 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:07.998 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:07.998 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:07.998 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:07.998 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:07.999 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:08.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:40:08.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T01:40:08.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:08.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:08.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:08.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:08.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:08.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:08.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:08.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:08.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:08.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:08.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:08.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:08 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:40:08.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:08 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:40:08.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:08 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi003 2023-07-20T01:40:08.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:08 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi003 to 5724M 2023-07-20T01:40:08.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:08 smithi003 ceph-mon[96422]: osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132] boot 2023-07-20T01:40:08.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:08 smithi003 ceph-mon[96422]: osdmap e44: 7 total, 7 up, 7 in 2023-07-20T01:40:08.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:08.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:08 smithi003 ceph-mon[96422]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:09.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:08 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:40:09.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:08 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:40:09.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:08 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi003 2023-07-20T01:40:09.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:08 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi003 to 5724M 2023-07-20T01:40:09.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:08 smithi043 ceph-mon[97823]: osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132] boot 2023-07-20T01:40:09.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:08 smithi043 ceph-mon[97823]: osdmap e44: 7 total, 7 up, 7 in 2023-07-20T01:40:09.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:08 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:09.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:08 smithi043 ceph-mon[97823]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.016+0000 7fb922b9a700 1 -- 172.21.15.43:0/3611042955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb91c0ff700 msgr2=0x7fb91c0ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.016+0000 7fb922b9a700 1 --2- 172.21.15.43:0/3611042955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb91c0ff700 0x7fb91c0ffb20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb9100099c0 tx=0x7fb910009cd0 comp rx=0 tx=0).stop 2023-07-20T01:40:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.016+0000 7fb922b9a700 1 -- 172.21.15.43:0/3611042955 shutdown_connections 2023-07-20T01:40:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.016+0000 7fb922b9a700 1 --2- 172.21.15.43:0/3611042955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91c1012a0 0x7fb91c107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.016+0000 7fb922b9a700 1 --2- 172.21.15.43:0/3611042955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb91c0ff700 0x7fb91c0ffb20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.016+0000 7fb922b9a700 1 --2- 172.21.15.43:0/3611042955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91c100900 0x7fb91c100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:09.033 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.016+0000 7fb922b9a700 1 -- 172.21.15.43:0/3611042955 >> 172.21.15.43:0/3611042955 conn(0x7fb91c0fac80 msgr2=0x7fb91c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:40:09.033 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.016+0000 7fb922b9a700 1 -- 172.21.15.43:0/3611042955 shutdown_connections 2023-07-20T01:40:09.033 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.016+0000 7fb922b9a700 1 -- 172.21.15.43:0/3611042955 wait complete. 2023-07-20T01:40:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb922b9a700 1 Processor -- start 2023-07-20T01:40:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb922b9a700 1 -- start start 2023-07-20T01:40:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb922b9a700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb91c0ff700 0x7fb91c1964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb922b9a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91c100900 0x7fb91c196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb922b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91c1012a0 0x7fb91c19dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb922b9a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c19e080 con 0x7fb91c1012a0 2023-07-20T01:40:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb922b9a700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c19e1c0 con 0x7fb91c0ff700 2023-07-20T01:40:09.036 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb922b9a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c19e350 con 0x7fb91c100900 2023-07-20T01:40:09.036 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb920936700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb91c0ff700 0x7fb91c1964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:09.036 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb920936700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb91c0ff700 0x7fb91c1964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48316/0 (socket says 172.21.15.43:48316) 2023-07-20T01:40:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb920936700 1 -- 172.21.15.43:0/298070826 learned_addr learned my addr 172.21.15.43:0/298070826 (peer_addr_for_me v2:172.21.15.43:0/0) 2023-07-20T01:40:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb91bfff700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91c100900 0x7fb91c196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb920936700 1 -- 172.21.15.43:0/298070826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91c100900 msgr2=0x7fb91c196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb920936700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91c100900 0x7fb91c196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:09.038 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb921137700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91c1012a0 0x7fb91c19dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:09.038 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb920936700 1 -- 172.21.15.43:0/298070826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91c1012a0 msgr2=0x7fb91c19dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:09.038 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb920936700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91c1012a0 0x7fb91c19dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb920936700 1 -- 172.21.15.43:0/298070826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9100095f0 con 0x7fb91c0ff700 2023-07-20T01:40:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.017+0000 7fb91bfff700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91c100900 0x7fb91c196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:40:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.018+0000 7fb920936700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb91c0ff700 0x7fb91c1964f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb910009990 tx=0x7fb910005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:40:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.018+0000 7fb919ffb700 1 -- 172.21.15.43:0/298070826 <== mon.2 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91001d070 con 0x7fb91c0ff700 2023-07-20T01:40:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.018+0000 7fb919ffb700 1 -- 172.21.15.43:0/298070826 <== mon.2 v2:172.21.15.43:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb910004200 con 0x7fb91c0ff700 2023-07-20T01:40:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.018+0000 7fb919ffb700 1 -- 172.21.15.43:0/298070826 <== mon.2 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910022e60 con 0x7fb91c0ff700 2023-07-20T01:40:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.018+0000 7fb922b9a700 1 -- 172.21.15.43:0/298070826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb91c19e5d0 con 0x7fb91c0ff700 2023-07-20T01:40:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.018+0000 7fb922b9a700 1 -- 172.21.15.43:0/298070826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb91c19ea70 con 0x7fb91c0ff700 2023-07-20T01:40:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.019+0000 7fb922b9a700 1 -- 172.21.15.43:0/298070826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91c106690 con 0x7fb91c0ff700 2023-07-20T01:40:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.020+0000 7fb919ffb700 1 -- 172.21.15.43:0/298070826 <== mon.2 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fb910022470 con 0x7fb91c0ff700 2023-07-20T01:40:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.020+0000 7fb919ffb700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb90c0a11d0 0x7fb90c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.021+0000 7fb919ffb700 1 -- 172.21.15.43:0/298070826 <== mon.2 v2:172.21.15.43:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fb9100cce00 con 0x7fb91c0ff700 2023-07-20T01:40:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.021+0000 7fb91bfff700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb90c0a11d0 0x7fb90c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.021+0000 7fb91bfff700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb90c0a11d0 0x7fb90c0a3690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb908005950 tx=0x7fb9080058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:40:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.025+0000 7fb919ffb700 1 -- 172.21.15.43:0/298070826 <== mon.2 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb91005c780 con 0x7fb91c0ff700 2023-07-20T01:40:09.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:08 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:40:09.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:08 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:40:09.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:08 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi003 2023-07-20T01:40:09.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:08 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi003 to 5724M 2023-07-20T01:40:09.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:08 smithi086 ceph-mon[98138]: osd.6 [v2:172.21.15.3:6818/1511361132,v1:172.21.15.3:6819/1511361132] boot 2023-07-20T01:40:09.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:08 smithi086 ceph-mon[98138]: osdmap e44: 7 total, 7 up, 7 in 2023-07-20T01:40:09.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:08 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:40:09.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:08 smithi086 ceph-mon[98138]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:09.198 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:09.229+0000 7fb922b9a700 1 -- 172.21.15.43:0/298070826 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb91c060bb0 con 0x7fb90c0a11d0 2023-07-20T01:40:09.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:09 smithi003 ceph-mon[96422]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T01:40:09.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:09 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:40:09.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:09 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:40:09.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:09 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:10.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:09 smithi043 ceph-mon[97823]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T01:40:10.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:09 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:40:10.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:09 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:40:10.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:09 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:10.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:09 smithi086 ceph-mon[98138]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T01:40:10.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:09 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:40:10.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:09 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:40:10.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:09 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:10.918 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:10 smithi043 ceph-mon[97823]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:40:10.918 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:10 smithi043 ceph-mon[97823]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:10.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:10 smithi003 ceph-mon[96422]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:40:10.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:10 smithi003 ceph-mon[96422]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:11.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:10 smithi086 ceph-mon[98138]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:40:11.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:10 smithi086 ceph-mon[98138]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:11.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:11 smithi003 ceph-mon[96422]: from='client.? 172.21.15.43:0/1352632361' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe6f3c63-9935-43b0-858f-11fd5f91c9ff"}]: dispatch 2023-07-20T01:40:11.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:11 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe6f3c63-9935-43b0-858f-11fd5f91c9ff"}]: dispatch 2023-07-20T01:40:11.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:11 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe6f3c63-9935-43b0-858f-11fd5f91c9ff"}]': finished 2023-07-20T01:40:11.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:11 smithi003 ceph-mon[96422]: osdmap e46: 8 total, 7 up, 8 in 2023-07-20T01:40:11.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:12.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:11 smithi043 ceph-mon[97823]: from='client.? 172.21.15.43:0/1352632361' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe6f3c63-9935-43b0-858f-11fd5f91c9ff"}]: dispatch 2023-07-20T01:40:12.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:11 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe6f3c63-9935-43b0-858f-11fd5f91c9ff"}]: dispatch 2023-07-20T01:40:12.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:11 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe6f3c63-9935-43b0-858f-11fd5f91c9ff"}]': finished 2023-07-20T01:40:12.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:11 smithi043 ceph-mon[97823]: osdmap e46: 8 total, 7 up, 8 in 2023-07-20T01:40:12.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:12.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:11 smithi086 ceph-mon[98138]: from='client.? 172.21.15.43:0/1352632361' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe6f3c63-9935-43b0-858f-11fd5f91c9ff"}]: dispatch 2023-07-20T01:40:12.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:11 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe6f3c63-9935-43b0-858f-11fd5f91c9ff"}]: dispatch 2023-07-20T01:40:12.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:11 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe6f3c63-9935-43b0-858f-11fd5f91c9ff"}]': finished 2023-07-20T01:40:12.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:11 smithi086 ceph-mon[98138]: osdmap e46: 8 total, 7 up, 8 in 2023-07-20T01:40:12.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:12.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:12 smithi003 ceph-mon[96422]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:12.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:12 smithi003 ceph-mon[96422]: from='client.? 172.21.15.43:0/3991224300' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:40:13.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:12 smithi043 ceph-mon[97823]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:13.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:12 smithi043 ceph-mon[97823]: from='client.? 172.21.15.43:0/3991224300' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:40:13.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:12 smithi086 ceph-mon[98138]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:13.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:12 smithi086 ceph-mon[98138]: from='client.? 172.21.15.43:0/3991224300' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:40:14.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:13 smithi003 ceph-mon[96422]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:14.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:13 smithi043 ceph-mon[97823]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:14.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:13 smithi086 ceph-mon[98138]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:16.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:15 smithi043 ceph-mon[97823]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:16.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:15 smithi003 ceph-mon[96422]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:16.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:15 smithi086 ceph-mon[98138]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:18.338 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:18 smithi043 ceph-mon[97823]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:18.338 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T01:40:18.338 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:18.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:18 smithi086 ceph-mon[98138]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:18.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T01:40:18.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:18.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:18 smithi003 ceph-mon[96422]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:18.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T01:40:18.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:19.595 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:19 smithi043 ceph-mon[97823]: Deploying daemon osd.7 on smithi043 2023-07-20T01:40:19.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:19 smithi086 ceph-mon[98138]: Deploying daemon osd.7 on smithi043 2023-07-20T01:40:19.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:19 smithi003 ceph-mon[96422]: Deploying daemon osd.7 on smithi043 2023-07-20T01:40:20.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:20 smithi086 ceph-mon[98138]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:20.627 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:20 smithi043 ceph-mon[97823]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:20.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:20 smithi003 ceph-mon[96422]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:22.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:21 smithi043 ceph-mon[97823]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:22.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:21 smithi003 ceph-mon[96422]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:22.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:21 smithi086 ceph-mon[98138]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:23.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:23.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:23.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:23.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:23.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:23.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:24.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:24.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:24.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:24.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:24 smithi043 ceph-mon[97823]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:24.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:24 smithi003 ceph-mon[96422]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:25.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:24 smithi086 ceph-mon[98138]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:25.462 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 7 on host 'smithi043' 2023-07-20T01:40:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.493+0000 7fb919ffb700 1 -- 172.21.15.43:0/298070826 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb91c060bb0 con 0x7fb90c0a11d0 2023-07-20T01:40:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 -- 172.21.15.43:0/298070826 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb90c0a11d0 msgr2=0x7fb90c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb90c0a11d0 0x7fb90c0a3690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb908005950 tx=0x7fb9080058e0 comp rx=0 tx=0).stop 2023-07-20T01:40:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 -- 172.21.15.43:0/298070826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb91c0ff700 msgr2=0x7fb91c1964f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb91c0ff700 0x7fb91c1964f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb910009990 tx=0x7fb910005e80 comp rx=0 tx=0).stop 2023-07-20T01:40:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 -- 172.21.15.43:0/298070826 shutdown_connections 2023-07-20T01:40:25.465 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb91c1012a0 0x7fb91c19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:25.465 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb90c0a11d0 0x7fb90c0a3690 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:25.465 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb91c0ff700 0x7fb91c1964f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:25.465 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 --2- 172.21.15.43:0/298070826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91c100900 0x7fb91c196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:25.465 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 -- 172.21.15.43:0/298070826 >> 172.21.15.43:0/298070826 conn(0x7fb91c0fac80 msgr2=0x7fb91c0fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:40:25.465 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 -- 172.21.15.43:0/298070826 shutdown_connections 2023-07-20T01:40:25.465 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:40:25.495+0000 7fb922b9a700 1 -- 172.21.15.43:0/298070826 wait complete. 2023-07-20T01:40:25.967 DEBUG:teuthology.orchestra.run.smithi043:osd.7> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.7.service 2023-07-20T01:40:25.969 INFO:tasks.cephadm:Deploying osd.8 on smithi086 with /dev/vg_nvme/lv_2... 2023-07-20T01:40:25.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T01:40:26.000 INFO:journalctl@ceph.osd.7.smithi043.stdout:-- Logs begin at Thu 2023-07-20 01:20:44 UTC. -- 2023-07-20T01:40:26.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:26.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:26.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:26 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:26.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:26 smithi043 ceph-mon[97823]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:26.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:26.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:26.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:26 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:26.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:26 smithi086 ceph-mon[98138]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:26.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:26.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:26.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:26.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:26 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:26.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:26 smithi003 ceph-mon[96422]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:26.545 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:40:26.729 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:40:26 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:40:26.636+0000 7fa3f5541640 -1 osd.7 0 log_to_monitors true 2023-07-20T01:40:27.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:27 smithi043 ceph-mon[97823]: from='osd.7 [v2:172.21.15.43:6816/3536213360,v1:172.21.15.43:6817/3536213360]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:40:27.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:27 smithi043 ceph-mon[97823]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:40:27.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:27 smithi086 ceph-mon[98138]: from='osd.7 [v2:172.21.15.43:6816/3536213360,v1:172.21.15.43:6817/3536213360]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:40:27.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:27 smithi086 ceph-mon[98138]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:40:27.884 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T01:40:27.884 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T01:40:27.884 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2023-07-20T01:40:27.884 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2023-07-20T01:40:27.885 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011132 s, 942 MB/s 2023-07-20T01:40:27.885 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2023-07-20T01:40:27.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:27 smithi003 ceph-mon[96422]: from='osd.7 [v2:172.21.15.43:6816/3536213360,v1:172.21.15.43:6817/3536213360]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:40:27.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:27 smithi003 ceph-mon[96422]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:40:28.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi086:vg_nvme/lv_2 2023-07-20T01:40:28.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T01:40:28.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T01:40:28.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:28.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='osd.7 [v2:172.21.15.43:6816/3536213360,v1:172.21.15.43:6817/3536213360]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:40:28.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:40:28.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:28.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:28.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:28.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:28.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:28.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:28.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:28.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:28.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:28.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:28 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:28.753 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:40:28.490+0000 7fa3ea3b8700 -1 osd.7 0 waiting for initial osdmap 2023-07-20T01:40:28.753 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:40:28.503+0000 7fa3e41aa700 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:40:28.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T01:40:28.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T01:40:28.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:28.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='osd.7 [v2:172.21.15.43:6816/3536213360,v1:172.21.15.43:6817/3536213360]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:40:28.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:40:28.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:28.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:28.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:28.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:28.756 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:28.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:28.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:28.757 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:28.758 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:28.758 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:28 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:28.834 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:40:28.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T01:40:28.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T01:40:28.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:28.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='osd.7 [v2:172.21.15.43:6816/3536213360,v1:172.21.15.43:6817/3536213360]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:40:28.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:40:28.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:28.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:28.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:28.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:28.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:28.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:28.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:28.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:28.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:28.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:28 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:29.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:29 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi043 2023-07-20T01:40:29.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:29 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi043 to 5724M 2023-07-20T01:40:29.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:29 smithi086 ceph-mon[98138]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:40:29.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:29 smithi086 ceph-mon[98138]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T01:40:29.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:29.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:29.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:29 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi043 2023-07-20T01:40:29.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:29 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi043 to 5724M 2023-07-20T01:40:29.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:29 smithi043 ceph-mon[97823]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:40:29.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:29 smithi043 ceph-mon[97823]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T01:40:29.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:29.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.926+0000 7fab1b58d700 1 -- 172.21.15.86:0/3393645278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab14100900 msgr2=0x7fab14100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.926+0000 7fab1b58d700 1 --2- 172.21.15.86:0/3393645278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab14100900 0x7fab14100d60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab10009990 tx=0x7fab10009ca0 comp rx=0 tx=0).stop 2023-07-20T01:40:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.926+0000 7fab1b58d700 1 -- 172.21.15.86:0/3393645278 shutdown_connections 2023-07-20T01:40:29.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.926+0000 7fab1b58d700 1 --2- 172.21.15.86:0/3393645278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab141012a0 0x7fab14107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:29.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.926+0000 7fab1b58d700 1 --2- 172.21.15.86:0/3393645278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab140ff700 0x7fab140ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:29.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.926+0000 7fab1b58d700 1 --2- 172.21.15.86:0/3393645278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab14100900 0x7fab14100d60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:29.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.926+0000 7fab1b58d700 1 -- 172.21.15.86:0/3393645278 >> 172.21.15.86:0/3393645278 conn(0x7fab140fac80 msgr2=0x7fab140fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:40:29.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab1b58d700 1 -- 172.21.15.86:0/3393645278 shutdown_connections 2023-07-20T01:40:29.896 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab1b58d700 1 -- 172.21.15.86:0/3393645278 wait complete. 2023-07-20T01:40:29.898 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab1b58d700 1 Processor -- start 2023-07-20T01:40:29.899 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab1b58d700 1 -- start start 2023-07-20T01:40:29.899 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab1b58d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab140ff700 0x7fab14069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:29.899 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab1b58d700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab14100900 0x7fab1419b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:29.899 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab1b58d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab141012a0 0x7fab1419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:29.899 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab1b58d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1419df60 con 0x7fab140ff700 2023-07-20T01:40:29.899 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab1b58d700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1419e0d0 con 0x7fab14100900 2023-07-20T01:40:29.899 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab1b58d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1419e290 con 0x7fab141012a0 2023-07-20T01:40:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab19b2a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab141012a0 0x7fab1419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab19b2a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab141012a0 0x7fab1419d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58640/0 (socket says 172.21.15.86:58640) 2023-07-20T01:40:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.927+0000 7fab19b2a700 1 -- 172.21.15.86:0/2978956166 learned_addr learned my addr 172.21.15.86:0/2978956166 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-07-20T01:40:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab18b28700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab14100900 0x7fab1419b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab19329700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab140ff700 0x7fab14069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab19b2a700 1 -- 172.21.15.86:0/2978956166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab14100900 msgr2=0x7fab1419b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab19b2a700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab14100900 0x7fab1419b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:29.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab19b2a700 1 -- 172.21.15.86:0/2978956166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab140ff700 msgr2=0x7fab14069620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:29.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab19b2a700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab140ff700 0x7fab14069620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:29.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab19b2a700 1 -- 172.21.15.86:0/2978956166 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab100095f0 con 0x7fab141012a0 2023-07-20T01:40:29.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab19b2a700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab141012a0 0x7fab1419d7c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fab0c007d70 tx=0x7fab0c00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:40:29.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab0a7fc700 1 -- 172.21.15.86:0/2978956166 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c00ad40 con 0x7fab141012a0 2023-07-20T01:40:29.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab0a7fc700 1 -- 172.21.15.86:0/2978956166 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab0c0128e0 con 0x7fab141012a0 2023-07-20T01:40:29.901 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab1b58d700 1 -- 172.21.15.86:0/2978956166 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab14194fb0 con 0x7fab141012a0 2023-07-20T01:40:29.902 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab0a7fc700 1 -- 172.21.15.86:0/2978956166 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c0116b0 con 0x7fab141012a0 2023-07-20T01:40:29.902 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.928+0000 7fab1b58d700 1 -- 172.21.15.86:0/2978956166 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab14195460 con 0x7fab141012a0 2023-07-20T01:40:29.902 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.929+0000 7fab1b58d700 1 -- 172.21.15.86:0/2978956166 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab14106c50 con 0x7fab141012a0 2023-07-20T01:40:29.902 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.930+0000 7fab0a7fc700 1 -- 172.21.15.86:0/2978956166 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fab0c012420 con 0x7fab141012a0 2023-07-20T01:40:29.903 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.931+0000 7fab0a7fc700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab000a1150 0x7fab000a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:29.903 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.931+0000 7fab0a7fc700 1 -- 172.21.15.86:0/2978956166 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5430+0+0 (secure 0 0 0) 0x7fab0c0cc4f0 con 0x7fab141012a0 2023-07-20T01:40:29.903 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.931+0000 7fab19329700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab000a1150 0x7fab000a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:29.903 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.931+0000 7fab19329700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab000a1150 0x7fab000a3610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fab04005950 tx=0x7fab040058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:40:29.903 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:29.934+0000 7fab0a7fc700 1 -- 172.21.15.86:0/2978956166 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fab0c0966e0 con 0x7fab141012a0 2023-07-20T01:40:29.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:29 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi043 2023-07-20T01:40:29.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:29 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi043 to 5724M 2023-07-20T01:40:29.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:29 smithi003 ceph-mon[96422]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:40:29.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:29 smithi003 ceph-mon[96422]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T01:40:29.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:29.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:30.125+0000 7fab1b58d700 1 -- 172.21.15.86:0/2978956166 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fab14196970 con 0x7fab000a1150 2023-07-20T01:40:30.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:30 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:40:30.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:30 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:40:30.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:30 smithi043 ceph-mon[97823]: osd.7 [v2:172.21.15.43:6816/3536213360,v1:172.21.15.43:6817/3536213360] boot 2023-07-20T01:40:30.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:30 smithi043 ceph-mon[97823]: osdmap e49: 8 total, 8 up, 8 in 2023-07-20T01:40:30.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:30.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:30 smithi043 ceph-mon[97823]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:30.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:40:30.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:40:30.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:30.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:30 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:40:30.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:30 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:40:30.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:30 smithi086 ceph-mon[98138]: osd.7 [v2:172.21.15.43:6816/3536213360,v1:172.21.15.43:6817/3536213360] boot 2023-07-20T01:40:30.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:30 smithi086 ceph-mon[98138]: osdmap e49: 8 total, 8 up, 8 in 2023-07-20T01:40:30.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:30.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:30 smithi086 ceph-mon[98138]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:30.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:40:30.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:40:30.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:30.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:30 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:40:30.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:30 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:40:30.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:30 smithi003 ceph-mon[96422]: osd.7 [v2:172.21.15.43:6816/3536213360,v1:172.21.15.43:6817/3536213360] boot 2023-07-20T01:40:30.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:30 smithi003 ceph-mon[96422]: osdmap e49: 8 total, 8 up, 8 in 2023-07-20T01:40:30.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:40:30.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:30 smithi003 ceph-mon[96422]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:40:30.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:40:30.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:40:30.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:31.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:31 smithi043 ceph-mon[97823]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:40:31.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:31 smithi043 ceph-mon[97823]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T01:40:31.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:31 smithi043 ceph-mon[97823]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T01:40:31.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:31 smithi086 ceph-mon[98138]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:40:31.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:31 smithi086 ceph-mon[98138]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T01:40:31.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:31 smithi086 ceph-mon[98138]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T01:40:31.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:31 smithi003 ceph-mon[96422]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:40:31.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:31 smithi003 ceph-mon[96422]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T01:40:31.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:31 smithi003 ceph-mon[96422]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T01:40:32.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:32 smithi043 ceph-mon[97823]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:32.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:32 smithi043 ceph-mon[97823]: from='client.? 172.21.15.86:0/2159768784' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc"}]: dispatch 2023-07-20T01:40:32.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:32 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc"}]: dispatch 2023-07-20T01:40:32.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:32 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc"}]': finished 2023-07-20T01:40:32.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:32 smithi043 ceph-mon[97823]: osdmap e52: 9 total, 8 up, 9 in 2023-07-20T01:40:32.755 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:32.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:32 smithi086 ceph-mon[98138]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:32.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:32 smithi086 ceph-mon[98138]: from='client.? 172.21.15.86:0/2159768784' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc"}]: dispatch 2023-07-20T01:40:32.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:32 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc"}]: dispatch 2023-07-20T01:40:32.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:32 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc"}]': finished 2023-07-20T01:40:32.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:32 smithi086 ceph-mon[98138]: osdmap e52: 9 total, 8 up, 9 in 2023-07-20T01:40:32.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:32.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:32 smithi003 ceph-mon[96422]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:32.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:32 smithi003 ceph-mon[96422]: from='client.? 172.21.15.86:0/2159768784' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc"}]: dispatch 2023-07-20T01:40:32.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:32 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc"}]: dispatch 2023-07-20T01:40:32.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:32 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc"}]': finished 2023-07-20T01:40:32.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:32 smithi003 ceph-mon[96422]: osdmap e52: 9 total, 8 up, 9 in 2023-07-20T01:40:32.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:33.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:33 smithi043 ceph-mon[97823]: from='client.? 172.21.15.86:0/1515134546' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:40:33.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:33 smithi043 ceph-mon[97823]: osdmap e53: 9 total, 8 up, 9 in 2023-07-20T01:40:33.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:33.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:33 smithi086 ceph-mon[98138]: from='client.? 172.21.15.86:0/1515134546' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:40:33.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:33 smithi086 ceph-mon[98138]: osdmap e53: 9 total, 8 up, 9 in 2023-07-20T01:40:33.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:33.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:33 smithi003 ceph-mon[96422]: from='client.? 172.21.15.86:0/1515134546' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:40:33.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:33 smithi003 ceph-mon[96422]: osdmap e53: 9 total, 8 up, 9 in 2023-07-20T01:40:33.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:34.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:34 smithi043 ceph-mon[97823]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:34.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:34 smithi086 ceph-mon[98138]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:34.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:34 smithi003 ceph-mon[96422]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:36.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:35 smithi003 ceph-mon[96422]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:36.187 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:40:35 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:40:35.804+0000 7fbd98cf4700 -1 calc_pg_upmaps abort due to max <= 0 2023-07-20T01:40:36.222 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:35 smithi086 ceph-mon[98138]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:36.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:35 smithi043 ceph-mon[97823]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:38.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:38 smithi086 ceph-mon[98138]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:38.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:38 smithi003 ceph-mon[96422]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:38.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:38 smithi043 ceph-mon[97823]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:39.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:39 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T01:40:39.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:39 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:39.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:39 smithi086 ceph-mon[98138]: Deploying daemon osd.8 on smithi086 2023-07-20T01:40:39.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:39 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T01:40:39.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:39 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:39.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:39 smithi003 ceph-mon[96422]: Deploying daemon osd.8 on smithi086 2023-07-20T01:40:39.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:39 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T01:40:39.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:39 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:39.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:39 smithi043 ceph-mon[97823]: Deploying daemon osd.8 on smithi086 2023-07-20T01:40:40.442 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:40 smithi086 ceph-mon[98138]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:40.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:40 smithi003 ceph-mon[96422]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:40.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:40 smithi043 ceph-mon[97823]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:42.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:41 smithi003 ceph-mon[96422]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:42.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:41 smithi043 ceph-mon[97823]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:42.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:41 smithi086 ceph-mon[98138]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:43.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:43 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:43.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:43 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:43.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:43 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:43.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:43 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:43.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:43 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:43.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:43 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:43.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:43 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:43.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:43 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:43.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:43 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:44.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:44 smithi086 ceph-mon[98138]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:44.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:44 smithi003 ceph-mon[96422]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:44.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:44 smithi043 ceph-mon[97823]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:45.600 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.632+0000 7fab0a7fc700 1 -- 172.21.15.86:0/2978956166 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fab14196970 con 0x7fab000a1150 2023-07-20T01:40:45.601 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 8 on host 'smithi086' 2023-07-20T01:40:45.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.634+0000 7fab1b58d700 1 -- 172.21.15.86:0/2978956166 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab000a1150 msgr2=0x7fab000a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:45.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.634+0000 7fab1b58d700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab000a1150 0x7fab000a3610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fab04005950 tx=0x7fab040058e0 comp rx=0 tx=0).stop 2023-07-20T01:40:45.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.634+0000 7fab1b58d700 1 -- 172.21.15.86:0/2978956166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab141012a0 msgr2=0x7fab1419d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:45.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.634+0000 7fab1b58d700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab141012a0 0x7fab1419d7c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fab0c007d70 tx=0x7fab0c00a340 comp rx=0 tx=0).stop 2023-07-20T01:40:45.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.634+0000 7fab1b58d700 1 -- 172.21.15.86:0/2978956166 shutdown_connections 2023-07-20T01:40:45.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.634+0000 7fab1b58d700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab141012a0 0x7fab1419d7c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:45.603 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.634+0000 7fab1b58d700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab140ff700 0x7fab14069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:45.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.634+0000 7fab1b58d700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab000a1150 0x7fab000a3610 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:45.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.634+0000 7fab1b58d700 1 --2- 172.21.15.86:0/2978956166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab14100900 0x7fab1419b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:45.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.634+0000 7fab1b58d700 1 -- 172.21.15.86:0/2978956166 >> 172.21.15.86:0/2978956166 conn(0x7fab140fac80 msgr2=0x7fab14103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:40:45.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.635+0000 7fab1b58d700 1 -- 172.21.15.86:0/2978956166 shutdown_connections 2023-07-20T01:40:45.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:40:45.635+0000 7fab1b58d700 1 -- 172.21.15.86:0/2978956166 wait complete. 2023-07-20T01:40:45.919 DEBUG:teuthology.orchestra.run.smithi086:osd.8> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.8.service 2023-07-20T01:40:45.921 INFO:tasks.cephadm:Deploying osd.9 on smithi003 with /dev/vg_nvme/lv_1... 2023-07-20T01:40:45.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T01:40:45.950 INFO:journalctl@ceph.osd.8.smithi086.stdout:-- Logs begin at Thu 2023-07-20 01:20:49 UTC. -- 2023-07-20T01:40:46.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:46 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:46 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:46 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:46.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:46 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:46.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:46 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:46 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:46 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:46 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:46.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:46 smithi086 ceph-mon[98138]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:46.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:46 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:46 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:46 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:46.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:46 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:46.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:46 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:46 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:46 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:46 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:46.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:46 smithi003 ceph-mon[96422]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:46.502 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:40:46.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:46 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:46 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:46 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:46.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:46 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:46.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:46 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:46 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:46 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:46.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:46 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:40:46.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:46 smithi043 ceph-mon[97823]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:47.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:40:46 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:40:46.874+0000 7f6bc13a9640 -1 osd.8 0 log_to_monitors true 2023-07-20T01:40:47.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:47 smithi086 ceph-mon[98138]: from='osd.8 [v2:172.21.15.86:6816/2823536777,v1:172.21.15.86:6817/2823536777]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T01:40:47.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:47 smithi086 ceph-mon[98138]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T01:40:47.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:47 smithi003 ceph-mon[96422]: from='osd.8 [v2:172.21.15.86:6816/2823536777,v1:172.21.15.86:6817/2823536777]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T01:40:47.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:47 smithi003 ceph-mon[96422]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T01:40:48.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:47 smithi043 ceph-mon[97823]: from='osd.8 [v2:172.21.15.86:6816/2823536777,v1:172.21.15.86:6817/2823536777]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T01:40:48.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:47 smithi043 ceph-mon[97823]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T01:40:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T01:40:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T01:40:48.018 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2023-07-20T01:40:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2023-07-20T01:40:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0107391 s, 976 MB/s 2023-07-20T01:40:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2023-07-20T01:40:48.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi003:vg_nvme/lv_1 2023-07-20T01:40:48.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T01:40:48.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T01:40:48.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:48.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='osd.8 [v2:172.21.15.86:6816/2823536777,v1:172.21.15.86:6817/2823536777]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:40:48.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:40:48.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:48.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:48.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:48.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:48.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:48.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:48.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:48 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:48.853 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:40:48.625+0000 7f6bb7a23700 -1 osd.8 0 waiting for initial osdmap 2023-07-20T01:40:48.854 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:40:48.638+0000 7f6bb0813700 -1 osd.8 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:40:48.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T01:40:48.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T01:40:48.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:48.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='osd.8 [v2:172.21.15.86:6816/2823536777,v1:172.21.15.86:6817/2823536777]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:40:48.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:40:48.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:48.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:48.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:48.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:48.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:48.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:48.858 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:48.858 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:48.858 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:48.859 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:48 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:49.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T01:40:49.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T01:40:49.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:49.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='osd.8 [v2:172.21.15.86:6816/2823536777,v1:172.21.15.86:6817/2823536777]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:40:49.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:40:49.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:40:49.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:49.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:49.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:49.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:49.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:40:49.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:49.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:49.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:40:49.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:48 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:40:49.012 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:40:49.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:49 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi086 2023-07-20T01:40:49.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:49 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi086 to 5722M 2023-07-20T01:40:49.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:49 smithi086 ceph-mon[98138]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:40:49.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:49 smithi086 ceph-mon[98138]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T01:40:49.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:49 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:49.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:49 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:49.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:49 smithi086 ceph-mon[98138]: osd.8 [v2:172.21.15.86:6816/2823536777,v1:172.21.15.86:6817/2823536777] boot 2023-07-20T01:40:49.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:49 smithi086 ceph-mon[98138]: osdmap e56: 9 total, 9 up, 9 in 2023-07-20T01:40:49.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:49 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:49.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:49 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi086 2023-07-20T01:40:49.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:49 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi086 to 5722M 2023-07-20T01:40:49.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:49 smithi003 ceph-mon[96422]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:40:49.891 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:49 smithi003 ceph-mon[96422]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T01:40:49.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:49 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:49.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:49 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:49.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:49 smithi003 ceph-mon[96422]: osd.8 [v2:172.21.15.86:6816/2823536777,v1:172.21.15.86:6817/2823536777] boot 2023-07-20T01:40:49.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:49 smithi003 ceph-mon[96422]: osdmap e56: 9 total, 9 up, 9 in 2023-07-20T01:40:49.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:49 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:50.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:49 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi086 2023-07-20T01:40:50.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:49 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi086 to 5722M 2023-07-20T01:40:50.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:49 smithi043 ceph-mon[97823]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:40:50.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:49 smithi043 ceph-mon[97823]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T01:40:50.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:49 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:50.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:49 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:50.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:49 smithi043 ceph-mon[97823]: osd.8 [v2:172.21.15.86:6816/2823536777,v1:172.21.15.86:6817/2823536777] boot 2023-07-20T01:40:50.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:49 smithi043 ceph-mon[97823]: osdmap e56: 9 total, 9 up, 9 in 2023-07-20T01:40:50.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:49 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:40:50.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.121+0000 7faa3744f700 1 -- 172.21.15.3:0/3249451583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa301012a0 msgr2=0x7faa30107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:50.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.121+0000 7faa3744f700 1 --2- 172.21.15.3:0/3249451583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa301012a0 0x7faa30107b50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faa2800b0d0 tx=0x7faa2800b3e0 comp rx=0 tx=0).stop 2023-07-20T01:40:50.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.121+0000 7faa3744f700 1 -- 172.21.15.3:0/3249451583 shutdown_connections 2023-07-20T01:40:50.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.121+0000 7faa3744f700 1 --2- 172.21.15.3:0/3249451583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa301012a0 0x7faa30107b50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:50.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.121+0000 7faa3744f700 1 --2- 172.21.15.3:0/3249451583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa300ff700 0x7faa300ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:50.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.121+0000 7faa3744f700 1 --2- 172.21.15.3:0/3249451583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa30100900 0x7faa30100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:50.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.121+0000 7faa3744f700 1 -- 172.21.15.3:0/3249451583 >> 172.21.15.3:0/3249451583 conn(0x7faa300fac80 msgr2=0x7faa300fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:40:50.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.121+0000 7faa3744f700 1 -- 172.21.15.3:0/3249451583 shutdown_connections 2023-07-20T01:40:50.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.121+0000 7faa3744f700 1 -- 172.21.15.3:0/3249451583 wait complete. 2023-07-20T01:40:50.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.121+0000 7faa3744f700 1 Processor -- start 2023-07-20T01:40:50.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa3744f700 1 -- start start 2023-07-20T01:40:50.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa3744f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa300ff700 0x7faa30068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:50.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa3744f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa30100900 0x7faa3019b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:50.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa3744f700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa301012a0 0x7faa3019d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:50.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa3744f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3019e000 con 0x7faa300ff700 2023-07-20T01:40:50.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa3744f700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3019e170 con 0x7faa301012a0 2023-07-20T01:40:50.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa3744f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3019e330 con 0x7faa30100900 2023-07-20T01:40:50.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa351eb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa300ff700 0x7faa30068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:50.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa351eb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa300ff700 0x7faa30068db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42216/0 (socket says 172.21.15.3:42216) 2023-07-20T01:40:50.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa351eb700 1 -- 172.21.15.3:0/515805390 learned_addr learned my addr 172.21.15.3:0/515805390 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:40:50.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa349ea700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa30100900 0x7faa3019b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:50.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa351eb700 1 -- 172.21.15.3:0/515805390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa30100900 msgr2=0x7faa3019b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:50.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa351eb700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa30100900 0x7faa3019b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:50.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa359ec700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa301012a0 0x7faa3019d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:50.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa351eb700 1 -- 172.21.15.3:0/515805390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa301012a0 msgr2=0x7faa3019d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:40:50.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa351eb700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa301012a0 0x7faa3019d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:40:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa351eb700 1 -- 172.21.15.3:0/515805390 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa28009d00 con 0x7faa300ff700 2023-07-20T01:40:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa349ea700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa30100900 0x7faa3019b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:40:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa359ec700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa301012a0 0x7faa3019d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:40:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.122+0000 7faa351eb700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa300ff700 0x7faa30068db0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faa2400c990 tx=0x7faa2400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:40:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.123+0000 7faa227fc700 1 -- 172.21.15.3:0/515805390 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24007b00 con 0x7faa300ff700 2023-07-20T01:40:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.123+0000 7faa227fc700 1 -- 172.21.15.3:0/515805390 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa24007c60 con 0x7faa300ff700 2023-07-20T01:40:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.123+0000 7faa3744f700 1 -- 172.21.15.3:0/515805390 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa3019e610 con 0x7faa300ff700 2023-07-20T01:40:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.123+0000 7faa227fc700 1 -- 172.21.15.3:0/515805390 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24018660 con 0x7faa300ff700 2023-07-20T01:40:50.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.123+0000 7faa3744f700 1 -- 172.21.15.3:0/515805390 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa3019eb60 con 0x7faa300ff700 2023-07-20T01:40:50.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.124+0000 7faa3744f700 1 -- 172.21.15.3:0/515805390 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa30106bc0 con 0x7faa300ff700 2023-07-20T01:40:50.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.125+0000 7faa227fc700 1 -- 172.21.15.3:0/515805390 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7faa2400f450 con 0x7faa300ff700 2023-07-20T01:40:50.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.125+0000 7faa227fc700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faa1c0a11d0 0x7faa1c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:40:50.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.125+0000 7faa349ea700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faa1c0a11d0 0x7faa1c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:40:50.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.125+0000 7faa227fc700 1 -- 172.21.15.3:0/515805390 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5862+0+0 (secure 0 0 0) 0x7faa240cbf40 con 0x7faa300ff700 2023-07-20T01:40:50.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.126+0000 7faa349ea700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faa1c0a11d0 0x7faa1c0a3690 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7faa2c005fd0 tx=0x7faa2c005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:40:50.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.130+0000 7faa227fc700 1 -- 172.21.15.3:0/515805390 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faa240960d0 con 0x7faa300ff700 2023-07-20T01:40:50.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:40:50.335+0000 7faa3744f700 1 -- 172.21.15.3:0/515805390 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7faa30060bb0 con 0x7faa1c0a11d0 2023-07-20T01:40:50.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:50 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:40:50.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:50 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:40:50.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:50 smithi086 ceph-mon[98138]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:40:50.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:50 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:40:50.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:50 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:40:50.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:50 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:50.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:50 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:40:50.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:50 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:40:50.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:50 smithi003 ceph-mon[96422]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:40:50.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:50 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:40:50.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:50 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:40:50.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:50 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:51.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:50 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:40:51.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:50 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:40:51.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:50 smithi043 ceph-mon[97823]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:40:51.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:50 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:40:51.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:50 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:40:51.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:50 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:40:51.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:51 smithi003 ceph-mon[96422]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:40:51.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:51 smithi003 ceph-mon[96422]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T01:40:52.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:51 smithi043 ceph-mon[97823]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:40:52.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:51 smithi043 ceph-mon[97823]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T01:40:52.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:51 smithi086 ceph-mon[98138]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:40:52.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:51 smithi086 ceph-mon[98138]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T01:40:52.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:52 smithi003 ceph-mon[96422]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T01:40:52.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:52 smithi003 ceph-mon[96422]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:40:52.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:52 smithi003 ceph-mon[96422]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T01:40:53.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:52 smithi043 ceph-mon[97823]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T01:40:53.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:52 smithi043 ceph-mon[97823]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:40:53.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:52 smithi043 ceph-mon[97823]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T01:40:53.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:52 smithi086 ceph-mon[98138]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T01:40:53.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:52 smithi086 ceph-mon[98138]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:40:53.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:52 smithi086 ceph-mon[98138]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T01:40:53.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:53 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2238347609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d356aa99-85dc-4006-9441-e4f7d23d594f"}]: dispatch 2023-07-20T01:40:53.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:53 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2238347609' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d356aa99-85dc-4006-9441-e4f7d23d594f"}]': finished 2023-07-20T01:40:53.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:53 smithi003 ceph-mon[96422]: osdmap e60: 10 total, 9 up, 10 in 2023-07-20T01:40:53.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:53 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:40:53.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:53 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/4183952044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:40:54.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:53 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2238347609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d356aa99-85dc-4006-9441-e4f7d23d594f"}]: dispatch 2023-07-20T01:40:54.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:53 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2238347609' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d356aa99-85dc-4006-9441-e4f7d23d594f"}]': finished 2023-07-20T01:40:54.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:53 smithi043 ceph-mon[97823]: osdmap e60: 10 total, 9 up, 10 in 2023-07-20T01:40:54.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:53 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:40:54.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:53 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/4183952044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:40:54.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:53 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2238347609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d356aa99-85dc-4006-9441-e4f7d23d594f"}]: dispatch 2023-07-20T01:40:54.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:53 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2238347609' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d356aa99-85dc-4006-9441-e4f7d23d594f"}]': finished 2023-07-20T01:40:54.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:53 smithi086 ceph-mon[98138]: osdmap e60: 10 total, 9 up, 10 in 2023-07-20T01:40:54.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:53 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:40:54.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:53 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/4183952044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:40:54.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:54 smithi003 ceph-mon[96422]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:40:55.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:54 smithi043 ceph-mon[97823]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:40:55.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:54 smithi086 ceph-mon[98138]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:40:56.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:55 smithi003 ceph-mon[96422]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2023-07-20T01:40:56.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:55 smithi043 ceph-mon[97823]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2023-07-20T01:40:56.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:55 smithi086 ceph-mon[98138]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2023-07-20T01:40:58.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:58 smithi086 ceph-mon[98138]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-20T01:40:58.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:58 smithi003 ceph-mon[96422]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-20T01:40:58.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:58 smithi043 ceph-mon[97823]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-20T01:41:00.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:40:59 smithi003 ceph-mon[96422]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:41:00.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:40:59 smithi043 ceph-mon[97823]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:41:00.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:40:59 smithi086 ceph-mon[98138]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:41:01.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T01:41:01.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:00 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:01.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:00 smithi003 ceph-mon[96422]: Deploying daemon osd.9 on smithi003 2023-07-20T01:41:01.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T01:41:01.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:00 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:01.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:00 smithi043 ceph-mon[97823]: Deploying daemon osd.9 on smithi003 2023-07-20T01:41:01.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T01:41:01.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:00 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:01.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:00 smithi086 ceph-mon[98138]: Deploying daemon osd.9 on smithi003 2023-07-20T01:41:02.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:01 smithi003 ceph-mon[96422]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-20T01:41:02.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:01 smithi043 ceph-mon[97823]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-20T01:41:02.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:01 smithi086 ceph-mon[98138]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-20T01:41:04.146 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:03 smithi003 ceph-mon[96422]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-07-20T01:41:04.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:03 smithi043 ceph-mon[97823]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-07-20T01:41:04.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:03 smithi086 ceph-mon[98138]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-07-20T01:41:05.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:05.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:05.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:05.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:05.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:05.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:05.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:05.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:05.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:06.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:06 smithi003 ceph-mon[96422]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:41:06.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:06 smithi043 ceph-mon[97823]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:41:06.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:06 smithi086 ceph-mon[98138]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:41:07.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.285+0000 7faa227fc700 1 -- 172.21.15.3:0/515805390 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faa30060bb0 con 0x7faa1c0a11d0 2023-07-20T01:41:07.254 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 9 on host 'smithi003' 2023-07-20T01:41:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 -- 172.21.15.3:0/515805390 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faa1c0a11d0 msgr2=0x7faa1c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faa1c0a11d0 0x7faa1c0a3690 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7faa2c005fd0 tx=0x7faa2c005ee0 comp rx=0 tx=0).stop 2023-07-20T01:41:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 -- 172.21.15.3:0/515805390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa300ff700 msgr2=0x7faa30068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa300ff700 0x7faa30068db0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faa2400c990 tx=0x7faa2400cd50 comp rx=0 tx=0).stop 2023-07-20T01:41:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 -- 172.21.15.3:0/515805390 shutdown_connections 2023-07-20T01:41:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faa1c0a11d0 0x7faa1c0a3690 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa301012a0 0x7faa3019d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa300ff700 0x7faa30068db0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 --2- 172.21.15.3:0/515805390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa30100900 0x7faa3019b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 -- 172.21.15.3:0/515805390 >> 172.21.15.3:0/515805390 conn(0x7faa300fac80 msgr2=0x7faa30103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 -- 172.21.15.3:0/515805390 shutdown_connections 2023-07-20T01:41:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:07.288+0000 7faa3744f700 1 -- 172.21.15.3:0/515805390 wait complete. 2023-07-20T01:41:07.800 DEBUG:teuthology.orchestra.run.smithi003:osd.9> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.9.service 2023-07-20T01:41:07.802 INFO:tasks.cephadm:Deploying osd.10 on smithi043 with /dev/vg_nvme/lv_1... 2023-07-20T01:41:07.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T01:41:07.832 INFO:journalctl@ceph.osd.9.smithi003.stdout:-- Logs begin at Thu 2023-07-20 01:20:47 UTC. -- 2023-07-20T01:41:08.206 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:08 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:41:08.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:08 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:08 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:08 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:08 smithi043 ceph-mon[97823]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:41:08.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:08 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:08 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:08 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:08 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:08 smithi086 ceph-mon[98138]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:41:08.687 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:41:08 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:41:08.446+0000 7f0651ea4640 -1 osd.9 0 log_to_monitors true 2023-07-20T01:41:08.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:08.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:08 smithi003 ceph-mon[96422]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:41:09.602 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:09 smithi003 ceph-mon[96422]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T01:41:09.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:09 smithi086 ceph-mon[98138]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T01:41:09.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:09 smithi043 ceph-mon[97823]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T01:41:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T01:41:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T01:41:09.886 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2023-07-20T01:41:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2023-07-20T01:41:09.886 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00807387 s, 1.3 GB/s 2023-07-20T01:41:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2023-07-20T01:41:10.322 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi043:vg_nvme/lv_1 2023-07-20T01:41:10.437 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:41:10 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:41:10.387+0000 7f0646d1b700 -1 osd.9 0 waiting for initial osdmap 2023-07-20T01:41:10.437 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:41:10 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:41:10.400+0000 7f0642310700 -1 osd.9 62 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:41:10.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:10 smithi003 ceph-mon[96422]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T01:41:10.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:10 smithi003 ceph-mon[96422]: osdmap e61: 10 total, 9 up, 10 in 2023-07-20T01:41:10.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:10 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:10.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:10 smithi003 ceph-mon[96422]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:41:10.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:10 smithi003 ceph-mon[96422]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:41:10.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:10 smithi043 ceph-mon[97823]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T01:41:10.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:10 smithi043 ceph-mon[97823]: osdmap e61: 10 total, 9 up, 10 in 2023-07-20T01:41:10.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:10 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:10.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:10 smithi043 ceph-mon[97823]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:41:10.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:10 smithi043 ceph-mon[97823]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:41:10.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:10 smithi086 ceph-mon[98138]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T01:41:10.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:10 smithi086 ceph-mon[98138]: osdmap e61: 10 total, 9 up, 10 in 2023-07-20T01:41:10.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:10 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:10.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:10 smithi086 ceph-mon[98138]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-07-20T01:41:10.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:10 smithi086 ceph-mon[98138]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:41:10.881 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:41:11.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:41:11.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T01:41:11.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:11.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:11.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi003 2023-07-20T01:41:11.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi003 to 4293M 2023-07-20T01:41:11.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:11.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:11.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:11.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:11.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:11.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.702 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:41:11.703 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T01:41:11.703 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:11.703 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:11.704 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi003 2023-07-20T01:41:11.704 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.704 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.704 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.705 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.705 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.705 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.706 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi003 to 4293M 2023-07-20T01:41:11.706 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.706 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:11.706 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:11.707 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.707 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:11.707 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:11.707 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:11.708 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-07-20T01:41:11.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T01:41:11.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:11.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:11.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi003 2023-07-20T01:41:11.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:11.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi003 to 4293M 2023-07-20T01:41:11.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:11.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:11.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:11.859 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:11.859 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:11.859 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:11.860 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:12.104 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.131+0000 7f8ed59db700 1 -- 172.21.15.43:0/2294407777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed0100900 msgr2=0x7f8ed0100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:12.105 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.131+0000 7f8ed59db700 1 --2- 172.21.15.43:0/2294407777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed0100900 0x7f8ed0100d60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00099c0 tx=0x7f8ec0009cd0 comp rx=0 tx=0).stop 2023-07-20T01:41:12.105 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.131+0000 7f8ed59db700 1 -- 172.21.15.43:0/2294407777 shutdown_connections 2023-07-20T01:41:12.105 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.131+0000 7f8ed59db700 1 --2- 172.21.15.43:0/2294407777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed01012a0 0x7f8ed0107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:12.105 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.131+0000 7f8ed59db700 1 --2- 172.21.15.43:0/2294407777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ed00ff700 0x7f8ed00ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:12.106 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.131+0000 7f8ed59db700 1 --2- 172.21.15.43:0/2294407777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed0100900 0x7f8ed0100d60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:12.106 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.131+0000 7f8ed59db700 1 -- 172.21.15.43:0/2294407777 >> 172.21.15.43:0/2294407777 conn(0x7f8ed00fac80 msgr2=0x7f8ed00fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:12.106 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.131+0000 7f8ed59db700 1 -- 172.21.15.43:0/2294407777 shutdown_connections 2023-07-20T01:41:12.106 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.131+0000 7f8ed59db700 1 -- 172.21.15.43:0/2294407777 wait complete. 2023-07-20T01:41:12.106 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.131+0000 7f8ed59db700 1 Processor -- start 2023-07-20T01:41:12.106 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ed59db700 1 -- start start 2023-07-20T01:41:12.106 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ed59db700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed00ff700 0x7f8ed0068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:12.106 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ed59db700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ed0100900 0x7f8ed019b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:12.107 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ed59db700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed01012a0 0x7f8ed019d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:12.107 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ed59db700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed019e080 con 0x7f8ed0100900 2023-07-20T01:41:12.107 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ed59db700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed019e1f0 con 0x7f8ed01012a0 2023-07-20T01:41:12.107 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ed59db700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed019e3b0 con 0x7f8ed00ff700 2023-07-20T01:41:12.108 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ecf7fe700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed01012a0 0x7f8ed019d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:12.108 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ecf7fe700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed01012a0 0x7f8ed019d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48458/0 (socket says 172.21.15.43:48458) 2023-07-20T01:41:12.108 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ecf7fe700 1 -- 172.21.15.43:0/2786655730 learned_addr learned my addr 172.21.15.43:0/2786655730 (peer_addr_for_me v2:172.21.15.43:0/0) 2023-07-20T01:41:12.108 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ece7fc700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ed0100900 0x7f8ed019b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:12.108 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8eceffd700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed00ff700 0x7f8ed0068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:12.109 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ecf7fe700 1 -- 172.21.15.43:0/2786655730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed00ff700 msgr2=0x7f8ed0068d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:12.109 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ecf7fe700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed00ff700 0x7f8ed0068d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:12.109 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ecf7fe700 1 -- 172.21.15.43:0/2786655730 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ed0100900 msgr2=0x7f8ed019b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:12.109 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ecf7fe700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ed0100900 0x7f8ed019b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:12.109 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ecf7fe700 1 -- 172.21.15.43:0/2786655730 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec00095f0 con 0x7f8ed01012a0 2023-07-20T01:41:12.109 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.132+0000 7f8ecf7fe700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed01012a0 0x7f8ed019d8e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800ada0 tx=0x7f8ec800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.133+0000 7f8ed49d9700 1 -- 172.21.15.43:0/2786655730 <== mon.2 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec800f070 con 0x7f8ed01012a0 2023-07-20T01:41:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.133+0000 7f8ed49d9700 1 -- 172.21.15.43:0/2786655730 <== mon.2 v2:172.21.15.43:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ec8007b30 con 0x7f8ed01012a0 2023-07-20T01:41:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.133+0000 7f8ed59db700 1 -- 172.21.15.43:0/2786655730 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed019e690 con 0x7f8ed01012a0 2023-07-20T01:41:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.133+0000 7f8ed49d9700 1 -- 172.21.15.43:0/2786655730 <== mon.2 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec80053e0 con 0x7f8ed01012a0 2023-07-20T01:41:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.133+0000 7f8ed59db700 1 -- 172.21.15.43:0/2786655730 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed019eb60 con 0x7f8ed01012a0 2023-07-20T01:41:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.134+0000 7f8ed59db700 1 -- 172.21.15.43:0/2786655730 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed0106d70 con 0x7f8ed01012a0 2023-07-20T01:41:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.135+0000 7f8ed49d9700 1 -- 172.21.15.43:0/2786655730 <== mon.2 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8ec800c980 con 0x7f8ed01012a0 2023-07-20T01:41:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.136+0000 7f8ed49d9700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8ebc0a1150 0x7f8ebc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.136+0000 7f8ed49d9700 1 -- 172.21.15.43:0/2786655730 <== mon.2 v2:172.21.15.43:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f8ec80cb6b0 con 0x7f8ed01012a0 2023-07-20T01:41:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.136+0000 7f8eceffd700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8ebc0a1150 0x7f8ebc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.136+0000 7f8eceffd700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8ebc0a1150 0x7f8ebc0a3610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8005950 tx=0x7f8eb80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:12.112 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.140+0000 7f8ed49d9700 1 -- 172.21.15.43:0/2786655730 <== mon.2 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8ec8095690 con 0x7f8ed01012a0 2023-07-20T01:41:12.317 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:12.349+0000 7f8ed59db700 1 -- 172.21.15.43:0/2786655730 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ed0060bb0 con 0x7f8ebc0a1150 2023-07-20T01:41:12.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:12 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:41:12.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:12 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:41:12.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:12 smithi003 ceph-mon[96422]: osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110] boot 2023-07-20T01:41:12.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:12 smithi003 ceph-mon[96422]: osdmap e63: 10 total, 10 up, 10 in 2023-07-20T01:41:12.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:12 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:12.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:12 smithi003 ceph-mon[96422]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:41:12.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:12 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:41:12.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:12 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:41:12.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:12 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:12.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:12 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:41:12.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:12 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:41:12.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:12 smithi043 ceph-mon[97823]: osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110] boot 2023-07-20T01:41:12.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:12 smithi043 ceph-mon[97823]: osdmap e63: 10 total, 10 up, 10 in 2023-07-20T01:41:12.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:12 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:12.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:12 smithi043 ceph-mon[97823]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:41:12.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:12 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:41:12.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:12 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:41:12.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:12 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:12.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:12 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:41:12.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:12 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:41:12.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:12 smithi086 ceph-mon[98138]: osd.9 [v2:172.21.15.3:6826/272617110,v1:172.21.15.3:6827/272617110] boot 2023-07-20T01:41:12.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:12 smithi086 ceph-mon[98138]: osdmap e63: 10 total, 10 up, 10 in 2023-07-20T01:41:12.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:12 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:41:12.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:12 smithi086 ceph-mon[98138]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:41:12.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:12 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:41:12.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:12 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:41:12.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:12 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:13.665 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:13 smithi043 ceph-mon[97823]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:41:13.665 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:13 smithi043 ceph-mon[97823]: osdmap e64: 10 total, 10 up, 10 in 2023-07-20T01:41:13.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:13 smithi003 ceph-mon[96422]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:41:13.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:13 smithi003 ceph-mon[96422]: osdmap e64: 10 total, 10 up, 10 in 2023-07-20T01:41:13.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:13 smithi086 ceph-mon[98138]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:41:13.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:13 smithi086 ceph-mon[98138]: osdmap e64: 10 total, 10 up, 10 in 2023-07-20T01:41:14.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:14 smithi003 ceph-mon[96422]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:14.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:14 smithi043 ceph-mon[97823]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:14.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:14 smithi086 ceph-mon[98138]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:15.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:15 smithi003 ceph-mon[96422]: from='client.? 172.21.15.43:0/2576388319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8848e6f-3991-4755-8fd3-627c9ebcd823"}]: dispatch 2023-07-20T01:41:15.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:15 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8848e6f-3991-4755-8fd3-627c9ebcd823"}]: dispatch 2023-07-20T01:41:15.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:15 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8848e6f-3991-4755-8fd3-627c9ebcd823"}]': finished 2023-07-20T01:41:15.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:15 smithi003 ceph-mon[96422]: osdmap e65: 11 total, 10 up, 11 in 2023-07-20T01:41:15.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:15 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:15.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:15 smithi003 ceph-mon[96422]: from='client.? 172.21.15.43:0/3487492804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:41:15.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:15 smithi043 ceph-mon[97823]: from='client.? 172.21.15.43:0/2576388319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8848e6f-3991-4755-8fd3-627c9ebcd823"}]: dispatch 2023-07-20T01:41:15.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:15 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8848e6f-3991-4755-8fd3-627c9ebcd823"}]: dispatch 2023-07-20T01:41:15.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:15 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8848e6f-3991-4755-8fd3-627c9ebcd823"}]': finished 2023-07-20T01:41:15.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:15 smithi043 ceph-mon[97823]: osdmap e65: 11 total, 10 up, 11 in 2023-07-20T01:41:15.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:15 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:15.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:15 smithi043 ceph-mon[97823]: from='client.? 172.21.15.43:0/3487492804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:41:15.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:15 smithi086 ceph-mon[98138]: from='client.? 172.21.15.43:0/2576388319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8848e6f-3991-4755-8fd3-627c9ebcd823"}]: dispatch 2023-07-20T01:41:15.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:15 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8848e6f-3991-4755-8fd3-627c9ebcd823"}]: dispatch 2023-07-20T01:41:15.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:15 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8848e6f-3991-4755-8fd3-627c9ebcd823"}]': finished 2023-07-20T01:41:15.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:15 smithi086 ceph-mon[98138]: osdmap e65: 11 total, 10 up, 11 in 2023-07-20T01:41:15.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:15 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:15.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:15 smithi086 ceph-mon[98138]: from='client.? 172.21.15.43:0/3487492804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:41:16.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:16 smithi003 ceph-mon[96422]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:16.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:16 smithi043 ceph-mon[97823]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:16.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:16 smithi086 ceph-mon[98138]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:18.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:18 smithi086 ceph-mon[98138]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:18.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:18 smithi003 ceph-mon[96422]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:18.731 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:18 smithi043 ceph-mon[97823]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:20.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:19 smithi003 ceph-mon[96422]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:20.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:19 smithi043 ceph-mon[97823]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:20.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:19 smithi086 ceph-mon[98138]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:21.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:21 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T01:41:21.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:21 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:21.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:21 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T01:41:21.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:21 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:21.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:21 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T01:41:21.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:21 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:22.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:22 smithi043 ceph-mon[97823]: Deploying daemon osd.10 on smithi043 2023-07-20T01:41:22.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:22 smithi043 ceph-mon[97823]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:22.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:22 smithi086 ceph-mon[98138]: Deploying daemon osd.10 on smithi043 2023-07-20T01:41:22.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:22 smithi086 ceph-mon[98138]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:22.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:22 smithi003 ceph-mon[96422]: Deploying daemon osd.10 on smithi043 2023-07-20T01:41:22.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:22 smithi003 ceph-mon[96422]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:24.154 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:23 smithi043 ceph-mon[97823]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:24.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:23 smithi003 ceph-mon[96422]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:24.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:23 smithi086 ceph-mon[98138]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:26.128 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:25 smithi043 ceph-mon[97823]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:26.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:25 smithi003 ceph-mon[96422]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:26.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:25 smithi086 ceph-mon[98138]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:27.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:27 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:27.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:27 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:27.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:27 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:27.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:27.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:27.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:27.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:27.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:27.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:28.611 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:28 smithi043 ceph-mon[97823]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:28.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:28 smithi003 ceph-mon[96422]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:28.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:28 smithi086 ceph-mon[98138]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:28.892 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.924+0000 7f8ed49d9700 1 -- 172.21.15.43:0/2786655730 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f8ed0060bb0 con 0x7f8ebc0a1150 2023-07-20T01:41:28.892 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 10 on host 'smithi043' 2023-07-20T01:41:28.894 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 -- 172.21.15.43:0/2786655730 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8ebc0a1150 msgr2=0x7f8ebc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:28.894 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8ebc0a1150 0x7f8ebc0a3610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8005950 tx=0x7f8eb80058e0 comp rx=0 tx=0).stop 2023-07-20T01:41:28.894 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 -- 172.21.15.43:0/2786655730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed01012a0 msgr2=0x7f8ed019d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed01012a0 0x7f8ed019d8e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800ada0 tx=0x7f8ec800c330 comp rx=0 tx=0).stop 2023-07-20T01:41:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 -- 172.21.15.43:0/2786655730 shutdown_connections 2023-07-20T01:41:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8ebc0a1150 0x7f8ebc0a3610 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ed01012a0 0x7f8ed019d8e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed00ff700 0x7f8ed0068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 --2- 172.21.15.43:0/2786655730 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ed0100900 0x7f8ed019b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:28.895 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 -- 172.21.15.43:0/2786655730 >> 172.21.15.43:0/2786655730 conn(0x7f8ed00fac80 msgr2=0x7f8ed0103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:28.896 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 -- 172.21.15.43:0/2786655730 shutdown_connections 2023-07-20T01:41:28.896 INFO:teuthology.orchestra.run.smithi043.stderr:2023-07-20T01:41:28.926+0000 7f8ed59db700 1 -- 172.21.15.43:0/2786655730 wait complete. 2023-07-20T01:41:29.268 DEBUG:teuthology.orchestra.run.smithi043:osd.10> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.10.service 2023-07-20T01:41:29.271 INFO:tasks.cephadm:Deploying osd.11 on smithi086 with /dev/vg_nvme/lv_1... 2023-07-20T01:41:29.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T01:41:29.300 INFO:journalctl@ceph.osd.10.smithi043.stdout:-- Logs begin at Thu 2023-07-20 01:20:44 UTC. -- 2023-07-20T01:41:29.835 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:41:30.112 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.113 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.113 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.113 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:29 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.113 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:29 smithi043 ceph-mon[97823]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:30.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:29 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:29 smithi003 ceph-mon[96422]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:30.200 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.200 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.201 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.201 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:29 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:30.201 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:29 smithi086 ceph-mon[98138]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:30.408 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:41:30 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:41:30.143+0000 7f9145780640 -1 osd.10 0 log_to_monitors true 2023-07-20T01:41:31.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:30 smithi043 ceph-mon[97823]: from='osd.10 [v2:172.21.15.43:6824/161869649,v1:172.21.15.43:6825/161869649]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:41:31.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:30 smithi043 ceph-mon[97823]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:41:31.283 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T01:41:31.283 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T01:41:31.283 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2023-07-20T01:41:31.283 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2023-07-20T01:41:31.284 INFO:teuthology.orchestra.run.smithi086.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112384 s, 933 MB/s 2023-07-20T01:41:31.284 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2023-07-20T01:41:31.318 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:30 smithi086 ceph-mon[98138]: from='osd.10 [v2:172.21.15.43:6824/161869649,v1:172.21.15.43:6825/161869649]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:41:31.318 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:30 smithi086 ceph-mon[98138]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:41:31.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:30 smithi003 ceph-mon[96422]: from='osd.10 [v2:172.21.15.43:6824/161869649,v1:172.21.15.43:6825/161869649]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:41:31.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:30 smithi003 ceph-mon[96422]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:41:31.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi086:vg_nvme/lv_1 2023-07-20T01:41:32.147 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:41:32.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:41:32.022+0000 7f913a5f7700 -1 osd.10 0 waiting for initial osdmap 2023-07-20T01:41:32.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:41:32.036+0000 7f9132be6700 -1 osd.10 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:41:32.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T01:41:32.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: osdmap e66: 11 total, 10 up, 11 in 2023-07-20T01:41:32.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:32.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='osd.10 [v2:172.21.15.43:6824/161869649,v1:172.21.15.43:6825/161869649]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:41:32.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:41:32.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi043 2023-07-20T01:41:32.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi043 to 4293M 2023-07-20T01:41:32.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:32.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:32.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:32.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:32.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:32.255 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:32.256 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:32 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T01:41:32.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: osdmap e66: 11 total, 10 up, 11 in 2023-07-20T01:41:32.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:32.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='osd.10 [v2:172.21.15.43:6824/161869649,v1:172.21.15.43:6825/161869649]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:41:32.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:41:32.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi043 2023-07-20T01:41:32.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi043 to 4293M 2023-07-20T01:41:32.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:32.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:32.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:32.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:32.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:32.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:32.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:32 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T01:41:32.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: osdmap e66: 11 total, 10 up, 11 in 2023-07-20T01:41:32.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:32.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='osd.10 [v2:172.21.15.43:6824/161869649,v1:172.21.15.43:6825/161869649]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:41:32.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2023-07-20T01:41:32.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi043 2023-07-20T01:41:32.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:32.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi043 to 4293M 2023-07-20T01:41:32.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:32.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:32.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:32.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:41:32.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:32.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:32.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:32.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:32 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:33.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:33 smithi043 ceph-mon[97823]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:41:33.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:33 smithi043 ceph-mon[97823]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T01:41:33.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:33.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:33.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:33 smithi043 ceph-mon[97823]: osd.10 [v2:172.21.15.43:6824/161869649,v1:172.21.15.43:6825/161869649] boot 2023-07-20T01:41:33.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:33 smithi043 ceph-mon[97823]: osdmap e68: 11 total, 11 up, 11 in 2023-07-20T01:41:33.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:33.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.362+0000 7fd76da94700 1 -- 172.21.15.86:0/669191412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd768100800 msgr2=0x7fd768100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:33.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.362+0000 7fd76da94700 1 --2- 172.21.15.86:0/669191412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd768100800 0x7fd768100c20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009990 tx=0x7fd75c009ca0 comp rx=0 tx=0).stop 2023-07-20T01:41:33.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.362+0000 7fd76da94700 1 -- 172.21.15.86:0/669191412 shutdown_connections 2023-07-20T01:41:33.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.362+0000 7fd76da94700 1 --2- 172.21.15.86:0/669191412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7681023a0 0x7fd768108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:33.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.362+0000 7fd76da94700 1 --2- 172.21.15.86:0/669191412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd768100800 0x7fd768100c20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:33.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.362+0000 7fd76da94700 1 --2- 172.21.15.86:0/669191412 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd768101a00 0x7fd768101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:33.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.362+0000 7fd76da94700 1 -- 172.21.15.86:0/669191412 >> 172.21.15.86:0/669191412 conn(0x7fd7680fbd60 msgr2=0x7fd7680fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:33.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.362+0000 7fd76da94700 1 -- 172.21.15.86:0/669191412 shutdown_connections 2023-07-20T01:41:33.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.362+0000 7fd76da94700 1 -- 172.21.15.86:0/669191412 wait complete. 2023-07-20T01:41:33.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.362+0000 7fd76da94700 1 Processor -- start 2023-07-20T01:41:33.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd76da94700 1 -- start start 2023-07-20T01:41:33.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd76da94700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd768100800 0x7fd768196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:33.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd76da94700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd768101a00 0x7fd768196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:33.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd76da94700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7681023a0 0x7fd76819d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:33.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd76da94700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76819dfb0 con 0x7fd7681023a0 2023-07-20T01:41:33.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd76da94700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76819e0f0 con 0x7fd768101a00 2023-07-20T01:41:33.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd76da94700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76819e280 con 0x7fd768100800 2023-07-20T01:41:33.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd7677fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd768100800 0x7fd768196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:33.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd7677fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd768100800 0x7fd768196420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58780/0 (socket says 172.21.15.86:58780) 2023-07-20T01:41:33.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd7677fe700 1 -- 172.21.15.86:0/1399171583 learned_addr learned my addr 172.21.15.86:0/1399171583 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-07-20T01:41:33.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd767fff700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7681023a0 0x7fd76819d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:33.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd766ffd700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd768101a00 0x7fd768196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:33.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd7677fe700 1 -- 172.21.15.86:0/1399171583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd768101a00 msgr2=0x7fd768196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:33.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd7677fe700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd768101a00 0x7fd768196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:33.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd7677fe700 1 -- 172.21.15.86:0/1399171583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7681023a0 msgr2=0x7fd76819d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:33.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd7677fe700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7681023a0 0x7fd76819d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:33.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd7677fe700 1 -- 172.21.15.86:0/1399171583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75c0095f0 con 0x7fd768100800 2023-07-20T01:41:33.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd767fff700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7681023a0 0x7fd76819d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:41:33.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd7677fe700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd768100800 0x7fd768196420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd75c003790 tx=0x7fd75c00f7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:33.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd764ff9700 1 -- 172.21.15.86:0/1399171583 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c01d070 con 0x7fd768100800 2023-07-20T01:41:33.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd76da94700 1 -- 172.21.15.86:0/1399171583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76819e500 con 0x7fd768100800 2023-07-20T01:41:33.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.363+0000 7fd764ff9700 1 -- 172.21.15.86:0/1399171583 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd75c003e20 con 0x7fd768100800 2023-07-20T01:41:33.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.364+0000 7fd764ff9700 1 -- 172.21.15.86:0/1399171583 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c017650 con 0x7fd768100800 2023-07-20T01:41:33.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.364+0000 7fd76da94700 1 -- 172.21.15.86:0/1399171583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76819e920 con 0x7fd768100800 2023-07-20T01:41:33.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.364+0000 7fd76da94700 1 -- 172.21.15.86:0/1399171583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76810c000 con 0x7fd768100800 2023-07-20T01:41:33.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.366+0000 7fd764ff9700 1 -- 172.21.15.86:0/1399171583 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fd75c003960 con 0x7fd768100800 2023-07-20T01:41:33.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.366+0000 7fd764ff9700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd7500a1150 0x7fd7500a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:33.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.366+0000 7fd764ff9700 1 -- 172.21.15.86:0/1399171583 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6727+0+0 (secure 0 0 0) 0x7fd75c0cd820 con 0x7fd768100800 2023-07-20T01:41:33.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.366+0000 7fd766ffd700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd7500a1150 0x7fd7500a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:33.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.367+0000 7fd766ffd700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd7500a1150 0x7fd7500a3610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd758005950 tx=0x7fd758005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:33.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.370+0000 7fd764ff9700 1 -- 172.21.15.86:0/1399171583 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd75c097630 con 0x7fd768100800 2023-07-20T01:41:33.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:33 smithi086 ceph-mon[98138]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:41:33.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:33 smithi086 ceph-mon[98138]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T01:41:33.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:33.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:33.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:33 smithi086 ceph-mon[98138]: osd.10 [v2:172.21.15.43:6824/161869649,v1:172.21.15.43:6825/161869649] boot 2023-07-20T01:41:33.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:33 smithi086 ceph-mon[98138]: osdmap e68: 11 total, 11 up, 11 in 2023-07-20T01:41:33.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:33.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:33 smithi003 ceph-mon[96422]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2023-07-20T01:41:33.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:33 smithi003 ceph-mon[96422]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T01:41:33.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:33.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:33.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:33 smithi003 ceph-mon[96422]: osd.10 [v2:172.21.15.43:6824/161869649,v1:172.21.15.43:6825/161869649] boot 2023-07-20T01:41:33.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:33 smithi003 ceph-mon[96422]: osdmap e68: 11 total, 11 up, 11 in 2023-07-20T01:41:33.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:41:33.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:33.564+0000 7fd76da94700 1 -- 172.21.15.86:0/1399171583 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd768060bb0 con 0x7fd7500a1150 2023-07-20T01:41:34.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:34 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:41:34.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:34 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:41:34.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:34 smithi043 ceph-mon[97823]: osdmap e69: 11 total, 11 up, 11 in 2023-07-20T01:41:34.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:34 smithi043 ceph-mon[97823]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:41:34.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:41:34.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:41:34.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:34 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:34.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:34 smithi043 ceph-mon[97823]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:34.326 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:34 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:41:34.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:34 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:41:34.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:34 smithi086 ceph-mon[98138]: osdmap e69: 11 total, 11 up, 11 in 2023-07-20T01:41:34.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:34 smithi086 ceph-mon[98138]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:41:34.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:41:34.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:41:34.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:34 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:34.329 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:34 smithi086 ceph-mon[98138]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:34.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:34 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:41:34.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:34 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:41:34.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:34 smithi003 ceph-mon[96422]: osdmap e69: 11 total, 11 up, 11 in 2023-07-20T01:41:34.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:34 smithi003 ceph-mon[96422]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:41:34.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:41:34.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:41:34.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:34 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:34.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:34 smithi003 ceph-mon[96422]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:35.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:35 smithi086 ceph-mon[98138]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T01:41:35.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:35 smithi003 ceph-mon[96422]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T01:41:35.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:35 smithi043 ceph-mon[97823]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T01:41:36.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:36 smithi003 ceph-mon[96422]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T01:41:36.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:36 smithi003 ceph-mon[96422]: pgmap v211: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 151 KiB/s, 0 objects/s recovering 2023-07-20T01:41:36.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:36 smithi003 ceph-mon[96422]: from='client.? 172.21.15.86:0/1828944895' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0fb9b4d-8559-48c2-9b41-c61a83ca0a44"}]: dispatch 2023-07-20T01:41:36.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:36 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0fb9b4d-8559-48c2-9b41-c61a83ca0a44"}]: dispatch 2023-07-20T01:41:36.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:36 smithi003 ceph-mon[96422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0fb9b4d-8559-48c2-9b41-c61a83ca0a44"}]': finished 2023-07-20T01:41:36.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:36 smithi003 ceph-mon[96422]: osdmap e72: 12 total, 11 up, 12 in 2023-07-20T01:41:36.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:36.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:36 smithi043 ceph-mon[97823]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T01:41:36.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:36 smithi043 ceph-mon[97823]: pgmap v211: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 151 KiB/s, 0 objects/s recovering 2023-07-20T01:41:36.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:36 smithi043 ceph-mon[97823]: from='client.? 172.21.15.86:0/1828944895' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0fb9b4d-8559-48c2-9b41-c61a83ca0a44"}]: dispatch 2023-07-20T01:41:36.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:36 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0fb9b4d-8559-48c2-9b41-c61a83ca0a44"}]: dispatch 2023-07-20T01:41:36.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:36 smithi043 ceph-mon[97823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0fb9b4d-8559-48c2-9b41-c61a83ca0a44"}]': finished 2023-07-20T01:41:36.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:36 smithi043 ceph-mon[97823]: osdmap e72: 12 total, 11 up, 12 in 2023-07-20T01:41:36.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:36.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:36 smithi086 ceph-mon[98138]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T01:41:36.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:36 smithi086 ceph-mon[98138]: pgmap v211: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 151 KiB/s, 0 objects/s recovering 2023-07-20T01:41:36.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:36 smithi086 ceph-mon[98138]: from='client.? 172.21.15.86:0/1828944895' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0fb9b4d-8559-48c2-9b41-c61a83ca0a44"}]: dispatch 2023-07-20T01:41:36.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:36 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0fb9b4d-8559-48c2-9b41-c61a83ca0a44"}]: dispatch 2023-07-20T01:41:36.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:36 smithi086 ceph-mon[98138]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0fb9b4d-8559-48c2-9b41-c61a83ca0a44"}]': finished 2023-07-20T01:41:36.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:36 smithi086 ceph-mon[98138]: osdmap e72: 12 total, 11 up, 12 in 2023-07-20T01:41:36.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:37.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:37 smithi003 ceph-mon[96422]: from='client.? 172.21.15.86:0/2537615306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:41:37.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:37 smithi043 ceph-mon[97823]: from='client.? 172.21.15.86:0/2537615306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:41:37.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:37 smithi086 ceph-mon[98138]: from='client.? 172.21.15.86:0/2537615306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:41:38.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:38 smithi003 ceph-mon[96422]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2023-07-20T01:41:38.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:38 smithi043 ceph-mon[97823]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2023-07-20T01:41:38.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:38 smithi086 ceph-mon[98138]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2023-07-20T01:41:40.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:39 smithi003 ceph-mon[96422]: pgmap v214: 1 pgs: 1 peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T01:41:40.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:39 smithi043 ceph-mon[97823]: pgmap v214: 1 pgs: 1 peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T01:41:40.273 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:39 smithi086 ceph-mon[98138]: pgmap v214: 1 pgs: 1 peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T01:41:42.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:41 smithi086 ceph-mon[98138]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-20T01:41:42.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:41 smithi003 ceph-mon[96422]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-20T01:41:42.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:41 smithi043 ceph-mon[97823]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-20T01:41:43.403 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:43 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T01:41:43.403 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:43 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:43.404 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:43 smithi086 ceph-mon[98138]: Deploying daemon osd.11 on smithi086 2023-07-20T01:41:43.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:43 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T01:41:43.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:43 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:43.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:43 smithi003 ceph-mon[96422]: Deploying daemon osd.11 on smithi086 2023-07-20T01:41:43.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:43 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T01:41:43.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:43 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:43.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:43 smithi043 ceph-mon[97823]: Deploying daemon osd.11 on smithi086 2023-07-20T01:41:44.578 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:44 smithi086 ceph-mon[98138]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-20T01:41:44.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:44 smithi003 ceph-mon[96422]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-20T01:41:44.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:44 smithi043 ceph-mon[97823]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-20T01:41:46.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:45 smithi003 ceph-mon[96422]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:46.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:45 smithi043 ceph-mon[97823]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:46.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:45 smithi086 ceph-mon[98138]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:47.961 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:47 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:47.961 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:47 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:47.961 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:47 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:48.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:48.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:48.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:47 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:48.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:47 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:48.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:47 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:48.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:47 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:49.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:48 smithi086 ceph-mon[98138]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:49.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:48 smithi003 ceph-mon[96422]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:49.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:48 smithi043 ceph-mon[97823]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:49.450 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.483+0000 7fd764ff9700 1 -- 172.21.15.86:0/1399171583 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fd768060bb0 con 0x7fd7500a1150 2023-07-20T01:41:49.452 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 11 on host 'smithi086' 2023-07-20T01:41:49.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 -- 172.21.15.86:0/1399171583 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd7500a1150 msgr2=0x7fd7500a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:49.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd7500a1150 0x7fd7500a3610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd758005950 tx=0x7fd758005cf0 comp rx=0 tx=0).stop 2023-07-20T01:41:49.454 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 -- 172.21.15.86:0/1399171583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd768100800 msgr2=0x7fd768196420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:49.454 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd768100800 0x7fd768196420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd75c003790 tx=0x7fd75c00f7b0 comp rx=0 tx=0).stop 2023-07-20T01:41:49.454 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 -- 172.21.15.86:0/1399171583 shutdown_connections 2023-07-20T01:41:49.455 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7681023a0 0x7fd76819d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:49.455 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd768100800 0x7fd768196420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:49.455 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd7500a1150 0x7fd7500a3610 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:49.456 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 --2- 172.21.15.86:0/1399171583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd768101a00 0x7fd768196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:49.456 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 -- 172.21.15.86:0/1399171583 >> 172.21.15.86:0/1399171583 conn(0x7fd7680fbd60 msgr2=0x7fd7680fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:49.457 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 -- 172.21.15.86:0/1399171583 shutdown_connections 2023-07-20T01:41:49.458 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:41:49.485+0000 7fd76da94700 1 -- 172.21.15.86:0/1399171583 wait complete. 2023-07-20T01:41:49.841 DEBUG:teuthology.orchestra.run.smithi086:osd.11> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.11.service 2023-07-20T01:41:49.847 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-20T01:41:49.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T01:41:49.876 INFO:journalctl@ceph.osd.11.smithi086.stdout:-- Logs begin at Thu 2023-07-20 01:20:49 UTC. -- 2023-07-20T01:41:50.427 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:41:50.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:50 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:50 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:50 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:50 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:50 smithi086 ceph-mon[98138]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:50.671 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:50 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.671 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:50 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.672 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:50 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.672 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:50 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.672 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:50 smithi003 ceph-mon[96422]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:50.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:50 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:50 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:50 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:50 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:50.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:50 smithi043 ceph-mon[97823]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:51.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:41:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:41:50.764+0000 7f3ee4d52640 -1 osd.11 0 log_to_monitors true 2023-07-20T01:41:51.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.620+0000 7f9661807700 1 -- 172.21.15.3:0/3741586565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f965c0ff700 msgr2=0x7f965c0ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:51.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.620+0000 7f9661807700 1 --2- 172.21.15.3:0/3741586565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f965c0ff700 0x7f965c0ffb20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f964c0099c0 tx=0x7f964c009cd0 comp rx=0 tx=0).stop 2023-07-20T01:41:51.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.620+0000 7f9661807700 1 -- 172.21.15.3:0/3741586565 shutdown_connections 2023-07-20T01:41:51.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.620+0000 7f9661807700 1 --2- 172.21.15.3:0/3741586565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f965c1012a0 0x7f965c107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:51.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.620+0000 7f9661807700 1 --2- 172.21.15.3:0/3741586565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f965c0ff700 0x7f965c0ffb20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:51.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.620+0000 7f9661807700 1 --2- 172.21.15.3:0/3741586565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f965c100900 0x7f965c100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:51.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.620+0000 7f9661807700 1 -- 172.21.15.3:0/3741586565 >> 172.21.15.3:0/3741586565 conn(0x7f965c0fac80 msgr2=0x7f965c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:51.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.620+0000 7f9661807700 1 -- 172.21.15.3:0/3741586565 shutdown_connections 2023-07-20T01:41:51.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.620+0000 7f9661807700 1 -- 172.21.15.3:0/3741586565 wait complete. 2023-07-20T01:41:51.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f9661807700 1 Processor -- start 2023-07-20T01:41:51.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f9661807700 1 -- start start 2023-07-20T01:41:51.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f9661807700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f965c0ff700 0x7f965c196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:51.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f9661807700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f965c100900 0x7f965c196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:51.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f9661807700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f965c1012a0 0x7f965c19dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:51.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f9661807700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c19e0a0 con 0x7f965c0ff700 2023-07-20T01:41:51.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f9661807700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c19e1e0 con 0x7f965c100900 2023-07-20T01:41:51.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f9661807700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c19e370 con 0x7f965c1012a0 2023-07-20T01:41:51.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f965affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f965c0ff700 0x7f965c196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:51.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f965affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f965c0ff700 0x7f965c196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42374/0 (socket says 172.21.15.3:42374) 2023-07-20T01:41:51.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f965b7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f965c1012a0 0x7f965c19dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:51.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.621+0000 7f965affd700 1 -- 172.21.15.3:0/3603897137 learned_addr learned my addr 172.21.15.3:0/3603897137 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:41:51.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f965a7fc700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f965c100900 0x7f965c196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:51.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f965b7fe700 1 -- 172.21.15.3:0/3603897137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f965c100900 msgr2=0x7f965c196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:51.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f965b7fe700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f965c100900 0x7f965c196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:51.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f965b7fe700 1 -- 172.21.15.3:0/3603897137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f965c0ff700 msgr2=0x7f965c196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:51.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f965b7fe700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f965c0ff700 0x7f965c196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:51.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f965b7fe700 1 -- 172.21.15.3:0/3603897137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c0095f0 con 0x7f965c1012a0 2023-07-20T01:41:51.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f965affd700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f965c0ff700 0x7f965c196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:41:51.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f965b7fe700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f965c1012a0 0x7f965c19dad0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9654007f40 tx=0x7f965400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:51.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f9660805700 1 -- 172.21.15.3:0/3603897137 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9654004db0 con 0x7f965c1012a0 2023-07-20T01:41:51.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f9661807700 1 -- 172.21.15.3:0/3603897137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965c19e650 con 0x7f965c1012a0 2023-07-20T01:41:51.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f9660805700 1 -- 172.21.15.3:0/3603897137 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f965400f040 con 0x7f965c1012a0 2023-07-20T01:41:51.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f9660805700 1 -- 172.21.15.3:0/3603897137 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96540059d0 con 0x7f965c1012a0 2023-07-20T01:41:51.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.622+0000 7f9661807700 1 -- 172.21.15.3:0/3603897137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965c19eba0 con 0x7f965c1012a0 2023-07-20T01:41:51.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.623+0000 7f9661807700 1 -- 172.21.15.3:0/3603897137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f965c1078c0 con 0x7f965c1012a0 2023-07-20T01:41:51.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.624+0000 7f9660805700 1 -- 172.21.15.3:0/3603897137 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f9654005000 con 0x7f965c1012a0 2023-07-20T01:41:51.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.625+0000 7f9660805700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f96440a1150 0x7f96440a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:51.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.625+0000 7f965affd700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f96440a1150 0x7f96440a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:51.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.625+0000 7f9660805700 1 -- 172.21.15.3:0/3603897137 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f96540cc4c0 con 0x7f965c1012a0 2023-07-20T01:41:51.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.626+0000 7f965affd700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f96440a1150 0x7f96440a3610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f964c004800 tx=0x7f964c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:51.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.631+0000 7f9660805700 1 -- 172.21.15.3:0/3603897137 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9654096140 con 0x7f965c1012a0 2023-07-20T01:41:51.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:51 smithi003 ceph-mon[96422]: from='osd.11 [v2:172.21.15.86:6824/3393537360,v1:172.21.15.86:6825/3393537360]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:41:51.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:51 smithi003 ceph-mon[96422]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:41:51.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.829+0000 7f9661807700 1 -- 172.21.15.3:0/3603897137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f965c02c970 con 0x7f965c1012a0 2023-07-20T01:41:51.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.830+0000 7f9660805700 1 -- 172.21.15.3:0/3603897137 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) v1 ==== 74+0+133 (secure 0 0 0) 0x7f9654099760 con 0x7f965c1012a0 2023-07-20T01:41:51.798 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:41:51.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 -- 172.21.15.3:0/3603897137 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f96440a1150 msgr2=0x7f96440a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:51.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f96440a1150 0x7f96440a3610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f964c004800 tx=0x7f964c01a040 comp rx=0 tx=0).stop 2023-07-20T01:41:51.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 -- 172.21.15.3:0/3603897137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f965c1012a0 msgr2=0x7f965c19dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:51.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f965c1012a0 0x7f965c19dad0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9654007f40 tx=0x7f965400d3b0 comp rx=0 tx=0).stop 2023-07-20T01:41:51.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 -- 172.21.15.3:0/3603897137 shutdown_connections 2023-07-20T01:41:51.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f965c1012a0 0x7f965c19dad0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:51.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f965c0ff700 0x7f965c196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:51.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f96440a1150 0x7f96440a3610 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:51.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 --2- 172.21.15.3:0/3603897137 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f965c100900 0x7f965c196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:51.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 -- 172.21.15.3:0/3603897137 >> 172.21.15.3:0/3603897137 conn(0x7f965c0fac80 msgr2=0x7f965c0fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:51.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 -- 172.21.15.3:0/3603897137 shutdown_connections 2023-07-20T01:41:51.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:51.832+0000 7f9661807700 1 -- 172.21.15.3:0/3603897137 wait complete. 2023-07-20T01:41:51.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:51 smithi086 ceph-mon[98138]: from='osd.11 [v2:172.21.15.86:6824/3393537360,v1:172.21.15.86:6825/3393537360]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:41:51.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:51 smithi086 ceph-mon[98138]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:41:52.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:51 smithi043 ceph-mon[97823]: from='osd.11 [v2:172.21.15.86:6824/3393537360,v1:172.21.15.86:6825/3393537360]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:41:52.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:51 smithi043 ceph-mon[97823]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:41:52.283 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":73,"num_osds":12,"num_up_osds":11,"osd_up_since":1689817292,"num_in_osds":12,"osd_in_since":1689817295,"num_remapped_pgs":0} 2023-07-20T01:41:52.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:41:52.553+0000 7f3ed9bc9700 -1 osd.11 0 waiting for initial osdmap 2023-07-20T01:41:52.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:41:52.567+0000 7f3ed39bb700 -1 osd.11 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:41:52.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T01:41:52.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: osdmap e73: 12 total, 11 up, 12 in 2023-07-20T01:41:52.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:52.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='osd.11 [v2:172.21.15.86:6824/3393537360,v1:172.21.15.86:6825/3393537360]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:41:52.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:41:52.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3603897137' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:41:52.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:52.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:52.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:52.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:52.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:52.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:52.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:52.855 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:52.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:52.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:52.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:52.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:52.856 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:52.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:52.857 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:52 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:52.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T01:41:52.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: osdmap e73: 12 total, 11 up, 12 in 2023-07-20T01:41:52.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:52.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='osd.11 [v2:172.21.15.86:6824/3393537360,v1:172.21.15.86:6825/3393537360]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:41:52.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:41:52.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3603897137' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:41:52.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:52.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:52.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:52.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:52.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:52.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:52.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:52.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:52.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:52.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:52.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:52.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:52.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:52.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:52.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:52 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:53.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T01:41:53.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: osdmap e73: 12 total, 11 up, 12 in 2023-07-20T01:41:53.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:53.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='osd.11 [v2:172.21.15.86:6824/3393537360,v1:172.21.15.86:6825/3393537360]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:41:53.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2023-07-20T01:41:53.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3603897137' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:41:53.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:53.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:53.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:53.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:53.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:53.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:53.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:41:53.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:53.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:53.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:53.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:53.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:41:53.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:41:53.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:41:53.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:52 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:41:53.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T01:41:53.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:53 smithi003 ceph-mon[96422]: Detected new or changed devices on smithi086 2023-07-20T01:41:53.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:53 smithi003 ceph-mon[96422]: Adjusting osd_memory_target on smithi086 to 4292M 2023-07-20T01:41:53.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:53 smithi003 ceph-mon[96422]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:41:53.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:53 smithi003 ceph-mon[96422]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T01:41:53.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:53 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:53.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:53 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:53.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:53 smithi086 ceph-mon[98138]: Detected new or changed devices on smithi086 2023-07-20T01:41:53.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:53 smithi086 ceph-mon[98138]: Adjusting osd_memory_target on smithi086 to 4292M 2023-07-20T01:41:53.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:53 smithi086 ceph-mon[98138]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:41:53.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:53 smithi086 ceph-mon[98138]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T01:41:53.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:53 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:53.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:53 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:53.862 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:41:54.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:53 smithi043 ceph-mon[97823]: Detected new or changed devices on smithi086 2023-07-20T01:41:54.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:53 smithi043 ceph-mon[97823]: Adjusting osd_memory_target on smithi086 to 4292M 2023-07-20T01:41:54.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:53 smithi043 ceph-mon[97823]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2023-07-20T01:41:54.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:53 smithi043 ceph-mon[97823]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T01:41:54.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:53 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:54.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:53 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:54.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:54 smithi003 ceph-mon[96422]: purged_snaps scrub starts 2023-07-20T01:41:54.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:54 smithi003 ceph-mon[96422]: purged_snaps scrub ok 2023-07-20T01:41:54.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:54 smithi003 ceph-mon[96422]: osd.11 [v2:172.21.15.86:6824/3393537360,v1:172.21.15.86:6825/3393537360] boot 2023-07-20T01:41:54.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:54 smithi003 ceph-mon[96422]: osdmap e75: 12 total, 12 up, 12 in 2023-07-20T01:41:54.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:54 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:54.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:54 smithi003 ceph-mon[96422]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:54.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:54 smithi086 ceph-mon[98138]: purged_snaps scrub starts 2023-07-20T01:41:54.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:54 smithi086 ceph-mon[98138]: purged_snaps scrub ok 2023-07-20T01:41:54.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:54 smithi086 ceph-mon[98138]: osd.11 [v2:172.21.15.86:6824/3393537360,v1:172.21.15.86:6825/3393537360] boot 2023-07-20T01:41:54.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:54 smithi086 ceph-mon[98138]: osdmap e75: 12 total, 12 up, 12 in 2023-07-20T01:41:54.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:54 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:54.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:54 smithi086 ceph-mon[98138]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:54.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.020+0000 7f8b96c90700 1 -- 172.21.15.3:0/3374961230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b90100900 msgr2=0x7f8b90100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:54.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.020+0000 7f8b96c90700 1 --2- 172.21.15.3:0/3374961230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b90100900 0x7f8b90100d60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c0099c0 tx=0x7f8b7c009cd0 comp rx=0 tx=0).stop 2023-07-20T01:41:54.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.020+0000 7f8b96c90700 1 -- 172.21.15.3:0/3374961230 shutdown_connections 2023-07-20T01:41:54.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.020+0000 7f8b96c90700 1 --2- 172.21.15.3:0/3374961230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b901012a0 0x7f8b90107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:54.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.020+0000 7f8b96c90700 1 --2- 172.21.15.3:0/3374961230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b900ff700 0x7f8b900ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:54.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.020+0000 7f8b96c90700 1 --2- 172.21.15.3:0/3374961230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b90100900 0x7f8b90100d60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:54.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.020+0000 7f8b96c90700 1 -- 172.21.15.3:0/3374961230 >> 172.21.15.3:0/3374961230 conn(0x7f8b900fac80 msgr2=0x7f8b900fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:54.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.020+0000 7f8b96c90700 1 -- 172.21.15.3:0/3374961230 shutdown_connections 2023-07-20T01:41:54.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.020+0000 7f8b96c90700 1 -- 172.21.15.3:0/3374961230 wait complete. 2023-07-20T01:41:54.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b96c90700 1 Processor -- start 2023-07-20T01:41:54.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b96c90700 1 -- start start 2023-07-20T01:41:54.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b96c90700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b900ff700 0x7f8b90196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:54.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b96c90700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b90100900 0x7f8b90196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:54.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b96c90700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b901012a0 0x7f8b9019dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:54.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b96c90700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9019e090 con 0x7f8b901012a0 2023-07-20T01:41:54.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b96c90700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9019e1d0 con 0x7f8b90100900 2023-07-20T01:41:54.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b96c90700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9019e360 con 0x7f8b900ff700 2023-07-20T01:41:54.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b9522d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b901012a0 0x7f8b9019dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:54.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b9522d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b901012a0 0x7f8b9019dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42394/0 (socket says 172.21.15.3:42394) 2023-07-20T01:41:54.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b9522d700 1 -- 172.21.15.3:0/597509628 learned_addr learned my addr 172.21.15.3:0/597509628 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:41:54.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b8ffff700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b90100900 0x7f8b90196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:54.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.021+0000 7f8b94a2c700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b900ff700 0x7f8b90196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:54.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.022+0000 7f8b9522d700 1 -- 172.21.15.3:0/597509628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b900ff700 msgr2=0x7f8b90196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:54.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.022+0000 7f8b9522d700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b900ff700 0x7f8b90196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:54.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.022+0000 7f8b9522d700 1 -- 172.21.15.3:0/597509628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b90100900 msgr2=0x7f8b90196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:54.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.022+0000 7f8b9522d700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b90100900 0x7f8b90196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:54.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.022+0000 7f8b9522d700 1 -- 172.21.15.3:0/597509628 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c0095f0 con 0x7f8b901012a0 2023-07-20T01:41:54.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.022+0000 7f8b9522d700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b901012a0 0x7f8b9019dac0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8b88007da0 tx=0x7f8b8800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:54.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.022+0000 7f8b8dffb700 1 -- 172.21.15.3:0/597509628 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8800ac50 con 0x7f8b901012a0 2023-07-20T01:41:54.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.022+0000 7f8b96c90700 1 -- 172.21.15.3:0/597509628 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9019e640 con 0x7f8b901012a0 2023-07-20T01:41:54.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.022+0000 7f8b8dffb700 1 -- 172.21.15.3:0/597509628 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b8800adb0 con 0x7f8b901012a0 2023-07-20T01:41:54.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.022+0000 7f8b8dffb700 1 -- 172.21.15.3:0/597509628 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b880115c0 con 0x7f8b901012a0 2023-07-20T01:41:54.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.022+0000 7f8b96c90700 1 -- 172.21.15.3:0/597509628 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9019eaf0 con 0x7f8b901012a0 2023-07-20T01:41:55.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.023+0000 7f8b96c90700 1 -- 172.21.15.3:0/597509628 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9010aff0 con 0x7f8b901012a0 2023-07-20T01:41:55.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.024+0000 7f8b8dffb700 1 -- 172.21.15.3:0/597509628 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8b88012420 con 0x7f8b901012a0 2023-07-20T01:41:55.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.025+0000 7f8b8dffb700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8b800a1150 0x7f8b800a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:55.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.025+0000 7f8b8dffb700 1 -- 172.21.15.3:0/597509628 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8b880cc6f0 con 0x7f8b901012a0 2023-07-20T01:41:55.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.025+0000 7f8b94a2c700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8b800a1150 0x7f8b800a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:55.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.025+0000 7f8b94a2c700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8b800a1150 0x7f8b800a3610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8b90100760 tx=0x7f8b84009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:55.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.029+0000 7f8b8dffb700 1 -- 172.21.15.3:0/597509628 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8b880962e0 con 0x7f8b901012a0 2023-07-20T01:41:55.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:54 smithi043 ceph-mon[97823]: purged_snaps scrub starts 2023-07-20T01:41:55.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:54 smithi043 ceph-mon[97823]: purged_snaps scrub ok 2023-07-20T01:41:55.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:54 smithi043 ceph-mon[97823]: osd.11 [v2:172.21.15.86:6824/3393537360,v1:172.21.15.86:6825/3393537360] boot 2023-07-20T01:41:55.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:54 smithi043 ceph-mon[97823]: osdmap e75: 12 total, 12 up, 12 in 2023-07-20T01:41:55.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:54 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:41:55.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:54 smithi043 ceph-mon[97823]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:41:55.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.225+0000 7f8b96c90700 1 -- 172.21.15.3:0/597509628 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f8b9019f190 con 0x7f8b901012a0 2023-07-20T01:41:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.226+0000 7f8b8dffb700 1 -- 172.21.15.3:0/597509628 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7f8b88099900 con 0x7f8b901012a0 2023-07-20T01:41:55.194 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:41:55.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 -- 172.21.15.3:0/597509628 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8b800a1150 msgr2=0x7f8b800a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:55.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8b800a1150 0x7f8b800a3610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8b90100760 tx=0x7f8b84009190 comp rx=0 tx=0).stop 2023-07-20T01:41:55.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 -- 172.21.15.3:0/597509628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b901012a0 msgr2=0x7f8b9019dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:55.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b901012a0 0x7f8b9019dac0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8b88007da0 tx=0x7f8b8800a340 comp rx=0 tx=0).stop 2023-07-20T01:41:55.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 -- 172.21.15.3:0/597509628 shutdown_connections 2023-07-20T01:41:55.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b901012a0 0x7f8b9019dac0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:55.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b900ff700 0x7f8b90196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:55.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8b800a1150 0x7f8b800a3610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:55.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 --2- 172.21.15.3:0/597509628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b90100900 0x7f8b90196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:55.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 -- 172.21.15.3:0/597509628 >> 172.21.15.3:0/597509628 conn(0x7f8b900fac80 msgr2=0x7f8b900fb560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:55.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 -- 172.21.15.3:0/597509628 shutdown_connections 2023-07-20T01:41:55.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:55.228+0000 7f8b96c90700 1 -- 172.21.15.3:0/597509628 wait complete. 2023-07-20T01:41:55.626 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":76,"num_osds":12,"num_up_osds":12,"osd_up_since":1689817313,"num_in_osds":12,"osd_in_since":1689817295,"num_remapped_pgs":0} 2023-07-20T01:41:55.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph osd dump --format=json 2023-07-20T01:41:55.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:55 smithi086 ceph-mon[98138]: osdmap e76: 12 total, 12 up, 12 in 2023-07-20T01:41:55.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:55 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/597509628' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:41:55.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:55 smithi003 ceph-mon[96422]: osdmap e76: 12 total, 12 up, 12 in 2023-07-20T01:41:55.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:55 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/597509628' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:41:56.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:55 smithi043 ceph-mon[97823]: osdmap e76: 12 total, 12 up, 12 in 2023-07-20T01:41:56.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:55 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/597509628' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:41:56.185 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:41:56.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:56 smithi003 ceph-mon[96422]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:41:56.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:56 smithi086 ceph-mon[98138]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:41:57.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:56 smithi043 ceph-mon[97823]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:41:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.350+0000 7f6489381700 1 -- 172.21.15.3:0/2293284843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6484106790 msgr2=0x7f6484106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.350+0000 7f6489381700 1 --2- 172.21.15.3:0/2293284843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6484106790 0x7f6484106b70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6474009990 tx=0x7f6474009ca0 comp rx=0 tx=0).stop 2023-07-20T01:41:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.350+0000 7f6489381700 1 -- 172.21.15.3:0/2293284843 shutdown_connections 2023-07-20T01:41:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.350+0000 7f6489381700 1 --2- 172.21.15.3:0/2293284843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6484101150 0x7f648410d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.350+0000 7f6489381700 1 --2- 172.21.15.3:0/2293284843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6484106790 0x7f6484106b70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.350+0000 7f6489381700 1 --2- 172.21.15.3:0/2293284843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6484100790 0x7f6484100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.350+0000 7f6489381700 1 -- 172.21.15.3:0/2293284843 >> 172.21.15.3:0/2293284843 conn(0x7f64840fc260 msgr2=0x7f64840fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.350+0000 7f6489381700 1 -- 172.21.15.3:0/2293284843 shutdown_connections 2023-07-20T01:41:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.350+0000 7f6489381700 1 -- 172.21.15.3:0/2293284843 wait complete. 2023-07-20T01:41:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.351+0000 7f6489381700 1 Processor -- start 2023-07-20T01:41:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.351+0000 7f6489381700 1 -- start start 2023-07-20T01:41:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.351+0000 7f6489381700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6484100790 0x7f648419c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.351+0000 7f6489381700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6484101150 0x7f648419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.351+0000 7f6489381700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6484106790 0x7f64841966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.351+0000 7f6489381700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6484196d30 con 0x7f6484101150 2023-07-20T01:41:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.351+0000 7f6489381700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6484196ea0 con 0x7f6484106790 2023-07-20T01:41:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.351+0000 7f6489381700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6484197060 con 0x7f6484100790 2023-07-20T01:41:57.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.351+0000 7f64827fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6484101150 0x7f648419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:57.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f64827fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6484101150 0x7f648419cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42412/0 (socket says 172.21.15.3:42412) 2023-07-20T01:41:57.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f6482ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6484100790 0x7f648419c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:57.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f64827fc700 1 -- 172.21.15.3:0/2365510540 learned_addr learned my addr 172.21.15.3:0/2365510540 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:41:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f64837fe700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6484106790 0x7f64841966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f64827fc700 1 -- 172.21.15.3:0/2365510540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6484100790 msgr2=0x7f648419c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f64827fc700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6484100790 0x7f648419c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f64827fc700 1 -- 172.21.15.3:0/2365510540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6484106790 msgr2=0x7f64841966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f64827fc700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6484106790 0x7f64841966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f64827fc700 1 -- 172.21.15.3:0/2365510540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64740095f0 con 0x7f6484101150 2023-07-20T01:41:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f64827fc700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6484101150 0x7f648419cb80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f646c00c920 tx=0x7f646c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f647bfff700 1 -- 172.21.15.3:0/2365510540 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c004a30 con 0x7f6484101150 2023-07-20T01:41:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f647bfff700 1 -- 172.21.15.3:0/2365510540 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f646c0054e0 con 0x7f6484101150 2023-07-20T01:41:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.352+0000 7f647bfff700 1 -- 172.21.15.3:0/2365510540 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c007bd0 con 0x7f6484101150 2023-07-20T01:41:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.353+0000 7f6489381700 1 -- 172.21.15.3:0/2365510540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6484197340 con 0x7f6484101150 2023-07-20T01:41:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.353+0000 7f6489381700 1 -- 172.21.15.3:0/2365510540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6484072b70 con 0x7f6484101150 2023-07-20T01:41:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.354+0000 7f6489381700 1 -- 172.21.15.3:0/2365510540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648404e470 con 0x7f6484101150 2023-07-20T01:41:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.355+0000 7f647bfff700 1 -- 172.21.15.3:0/2365510540 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f646c007d50 con 0x7f6484101150 2023-07-20T01:41:57.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.355+0000 7f647bfff700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f64700a56b0 0x7f64700a7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:57.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.356+0000 7f6482ffd700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f64700a56b0 0x7f64700a7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:57.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.356+0000 7f647bfff700 1 -- 172.21.15.3:0/2365510540 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f646c0cbeb0 con 0x7f6484101150 2023-07-20T01:41:57.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.356+0000 7f6482ffd700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f64700a56b0 0x7f64700a7b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f647400b5c0 tx=0x7f6474005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:57.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.360+0000 7f647bfff700 1 -- 172.21.15.3:0/2365510540 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f646c095aa0 con 0x7f6484101150 2023-07-20T01:41:57.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.555+0000 7f6489381700 1 -- 172.21.15.3:0/2365510540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64841974d0 con 0x7f6484101150 2023-07-20T01:41:57.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.556+0000 7f647bfff700 1 -- 172.21.15.3:0/2365510540 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18240 (secure 0 0 0) 0x7f646c01d090 con 0x7f6484101150 2023-07-20T01:41:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:41:57.524 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":76,"fsid":"9b4fd2fc-269d-11ee-9b34-001a4aab830c","created":"2023-07-20T01:35:37.746283+0000","modified":"2023-07-20T01:41:54.557580+0000","last_up_change":"2023-07-20T01:41:53.551492+0000","last_in_change":"2023-07-20T01:41:35.988339+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T01:38:42.316631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0073d11-a48f-415c-babd-5672ef2585b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6803","nonce":1627689053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6805","nonce":1627689053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6809","nonce":1627689053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6807","nonce":1627689053}]},"public_addr":"172.21.15.3:6803/1627689053","cluster_addr":"172.21.15.3:6805/1627689053","heartbeat_back_addr":"172.21.15.3:6809/1627689053","heartbeat_front_addr":"172.21.15.3:6807/1627689053","state":["exists","up"]},{"osd":1,"uuid":"91f60b3d-e08c-4a41-b480-c3d30348eacc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6801","nonce":3632963249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6803","nonce":3632963249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6807","nonce":3632963249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6805","nonce":3632963249}]},"public_addr":"172.21.15.43:6801/3632963249","cluster_addr":"172.21.15.43:6803/3632963249","heartbeat_back_addr":"172.21.15.43:6807/3632963249","heartbeat_front_addr":"172.21.15.43:6805/3632963249","state":["exists","up"]},{"osd":2,"uuid":"11808f4b-c50a-44f4-8581-3fc226653b39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6801","nonce":2141120676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6803","nonce":2141120676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6807","nonce":2141120676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6805","nonce":2141120676}]},"public_addr":"172.21.15.86:6801/2141120676","cluster_addr":"172.21.15.86:6803/2141120676","heartbeat_back_addr":"172.21.15.86:6807/2141120676","heartbeat_front_addr":"172.21.15.86:6805/2141120676","state":["exists","up"]},{"osd":3,"uuid":"45c2d3f6-9f5d-4510-9dad-6d95d044725f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6811","nonce":535233193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6813","nonce":535233193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6817","nonce":535233193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6815","nonce":535233193}]},"public_addr":"172.21.15.3:6811/535233193","cluster_addr":"172.21.15.3:6813/535233193","heartbeat_back_addr":"172.21.15.3:6817/535233193","heartbeat_front_addr":"172.21.15.3:6815/535233193","state":["exists","up"]},{"osd":4,"uuid":"7a35bbae-6f6c-4296-9459-f1b4d2de632f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6809","nonce":47431618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6811","nonce":47431618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6815","nonce":47431618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6813","nonce":47431618}]},"public_addr":"172.21.15.43:6809/47431618","cluster_addr":"172.21.15.43:6811/47431618","heartbeat_back_addr":"172.21.15.43:6815/47431618","heartbeat_front_addr":"172.21.15.43:6813/47431618","state":["exists","up"]},{"osd":5,"uuid":"34cccdff-563b-413c-b56c-08839b1534aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6809","nonce":1573392100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1573392100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1573392100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1573392100}]},"public_addr":"172.21.15.86:6809/1573392100","cluster_addr":"172.21.15.86:6811/1573392100","heartbeat_back_addr":"172.21.15.86:6815/1573392100","heartbeat_front_addr":"172.21.15.86:6813/1573392100","state":["exists","up"]},{"osd":6,"uuid":"e1c8f2b2-aed6-4277-b645-6a68e287ca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6819","nonce":1511361132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6821","nonce":1511361132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6825","nonce":1511361132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6823","nonce":1511361132}]},"public_addr":"172.21.15.3:6819/1511361132","cluster_addr":"172.21.15.3:6821/1511361132","heartbeat_back_addr":"172.21.15.3:6825/1511361132","heartbeat_front_addr":"172.21.15.3:6823/1511361132","state":["exists","up"]},{"osd":7,"uuid":"fe6f3c63-9935-43b0-858f-11fd5f91c9ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6817","nonce":3536213360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6819","nonce":3536213360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6823","nonce":3536213360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6821","nonce":3536213360}]},"public_addr":"172.21.15.43:6817/3536213360","cluster_addr":"172.21.15.43:6819/3536213360","heartbeat_back_addr":"172.21.15.43:6823/3536213360","heartbeat_front_addr":"172.21.15.43:6821/3536213360","state":["exists","up"]},{"osd":8,"uuid":"7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6817","nonce":2823536777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6819","nonce":2823536777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6823","nonce":2823536777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6821","nonce":2823536777}]},"public_addr":"172.21.15.86:6817/2823536777","cluster_addr":"172.21.15.86:6819/2823536777","heartbeat_back_addr":"172.21.15.86:6823/2823536777","heartbeat_front_addr":"172.21.15.86:6821/2823536777","state":["exists","up"]},{"osd":9,"uuid":"d356aa99-85dc-4006-9441-e4f7d23d594f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6827","nonce":272617110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6829","nonce":272617110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6833","nonce":272617110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6831","nonce":272617110}]},"public_addr":"172.21.15.3:6827/272617110","cluster_addr":"172.21.15.3:6829/272617110","heartbeat_back_addr":"172.21.15.3:6833/272617110","heartbeat_front_addr":"172.21.15.3:6831/272617110","state":["exists","up"]},{"osd":10,"uuid":"b8848e6f-3991-4755-8fd3-627c9ebcd823","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6825","nonce":161869649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6827","nonce":161869649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6831","nonce":161869649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6829","nonce":161869649}]},"public_addr":"172.21.15.43:6825/161869649","cluster_addr":"172.21.15.43:6827/161869649","heartbeat_back_addr":"172.21.15.43:6831/161869649","heartbeat_front_addr":"172.21.15.43:6829/161869649","state":["exists","up"]},{"osd":11,"uuid":"e0fb9b4d-8559-48c2-9b41-c61a83ca0a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6825","nonce":3393537360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6827","nonce":3393537360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6831","nonce":3393537360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6829","nonce":3393537360}]},"public_addr":"172.21.15.86:6825/3393537360","cluster_addr":"172.21.15.86:6827/3393537360","heartbeat_back_addr":"172.21.15.86:6831/3393537360","heartbeat_front_addr":"172.21.15.86:6829/3393537360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:37:59.181659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:38:20.326807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:38:40.402049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:02.811605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:23.889505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:44.425791+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:06.236232+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:27.676116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:47.846169+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:09.462761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:31.165679+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:51.723548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/942763937":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6801/596448098":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6800/596448098":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6800/417953684":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/1023086410":"2023-07-21T01:36:35.772246+0000","172.21.15.3:0/3900831138":"2023-07-21T01:36:35.772246+0000","172.21.15.3:0/2570287892":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/286795203":"2023-07-21T01:36:10.337325+0000","172.21.15.3:6801/417953684":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/2757371565":"2023-07-21T01:36:10.337325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T01:41:57.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 -- 172.21.15.3:0/2365510540 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f64700a56b0 msgr2=0x7f64700a7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:57.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f64700a56b0 0x7f64700a7b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f647400b5c0 tx=0x7f6474005fd0 comp rx=0 tx=0).stop 2023-07-20T01:41:57.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 -- 172.21.15.3:0/2365510540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6484101150 msgr2=0x7f648419cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:57.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6484101150 0x7f648419cb80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f646c00c920 tx=0x7f646c00cc30 comp rx=0 tx=0).stop 2023-07-20T01:41:57.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 -- 172.21.15.3:0/2365510540 shutdown_connections 2023-07-20T01:41:57.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f64700a56b0 0x7f64700a7b70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:57.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6484106790 0x7f64841966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:57.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6484100790 0x7f648419c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:57.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 --2- 172.21.15.3:0/2365510540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6484101150 0x7f648419cb80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:57.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 -- 172.21.15.3:0/2365510540 >> 172.21.15.3:0/2365510540 conn(0x7f64840fc260 msgr2=0x7f64840fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:57.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 -- 172.21.15.3:0/2365510540 shutdown_connections 2023-07-20T01:41:57.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:57.558+0000 7f6489381700 1 -- 172.21.15.3:0/2365510540 wait complete. 2023-07-20T01:41:57.947 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T01:38:42.316631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T01:41:57.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-20T01:41:58.503 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:41:58.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:58 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2365510540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:41:58.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:58 smithi003 ceph-mon[96422]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:41:58.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:58 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2365510540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:41:58.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:58 smithi043 ceph-mon[97823]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:41:58.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:58 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2365510540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:41:58.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:58 smithi086 ceph-mon[98138]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:41:59.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.754+0000 7f3f830aa700 1 -- 172.21.15.3:0/275666022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c1051e0 msgr2=0x7f3f7c105660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:59.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.754+0000 7f3f830aa700 1 --2- 172.21.15.3:0/275666022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c1051e0 0x7f3f7c105660 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3f680099c0 tx=0x7f3f68009cd0 comp rx=0 tx=0).stop 2023-07-20T01:41:59.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.754+0000 7f3f830aa700 1 -- 172.21.15.3:0/275666022 shutdown_connections 2023-07-20T01:41:59.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.754+0000 7f3f830aa700 1 --2- 172.21.15.3:0/275666022 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f7c105ba0 0x7f3f7c10d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:59.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.754+0000 7f3f830aa700 1 --2- 172.21.15.3:0/275666022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f7c068730 0x7f3f7c068b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:59.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.754+0000 7f3f830aa700 1 --2- 172.21.15.3:0/275666022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c1051e0 0x7f3f7c105660 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:59.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.754+0000 7f3f830aa700 1 -- 172.21.15.3:0/275666022 >> 172.21.15.3:0/275666022 conn(0x7f3f7c076770 msgr2=0x7f3f7c076b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:59.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.754+0000 7f3f830aa700 1 -- 172.21.15.3:0/275666022 shutdown_connections 2023-07-20T01:41:59.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.755+0000 7f3f830aa700 1 -- 172.21.15.3:0/275666022 wait complete. 2023-07-20T01:41:59.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.755+0000 7f3f830aa700 1 Processor -- start 2023-07-20T01:41:59.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.755+0000 7f3f830aa700 1 -- start start 2023-07-20T01:41:59.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.755+0000 7f3f830aa700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f7c068730 0x7f3f7c19c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:59.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.755+0000 7f3f830aa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c1051e0 0x7f3f7c19cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:59.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.755+0000 7f3f830aa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f7c105ba0 0x7f3f7c196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:59.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f830aa700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c196cc0 con 0x7f3f7c1051e0 2023-07-20T01:41:59.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f830aa700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c196e30 con 0x7f3f7c068730 2023-07-20T01:41:59.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f830aa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c196ff0 con 0x7f3f7c105ba0 2023-07-20T01:41:59.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f7bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c1051e0 0x7f3f7c19cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:59.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f7bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c1051e0 0x7f3f7c19cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42424/0 (socket says 172.21.15.3:42424) 2023-07-20T01:41:59.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f7bfff700 1 -- 172.21.15.3:0/3946862751 learned_addr learned my addr 172.21.15.3:0/3946862751 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:41:59.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f81647700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f7c105ba0 0x7f3f7c196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:59.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f80e46700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f7c068730 0x7f3f7c19c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:59.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f7bfff700 1 -- 172.21.15.3:0/3946862751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f7c105ba0 msgr2=0x7f3f7c196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:59.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f7bfff700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f7c105ba0 0x7f3f7c196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:59.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f7bfff700 1 -- 172.21.15.3:0/3946862751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f7c068730 msgr2=0x7f3f7c19c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:59.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f7bfff700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f7c068730 0x7f3f7c19c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:59.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f7bfff700 1 -- 172.21.15.3:0/3946862751 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f680095f0 con 0x7f3f7c1051e0 2023-07-20T01:41:59.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f7bfff700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c1051e0 0x7f3f7c19cb10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3f68005910 tx=0x7f3f68003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:59.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f79ffb700 1 -- 172.21.15.3:0/3946862751 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6801d070 con 0x7f3f7c1051e0 2023-07-20T01:41:59.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f79ffb700 1 -- 172.21.15.3:0/3946862751 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f68022470 con 0x7f3f7c1051e0 2023-07-20T01:41:59.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f830aa700 1 -- 172.21.15.3:0/3946862751 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c197270 con 0x7f3f7c1051e0 2023-07-20T01:41:59.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.756+0000 7f3f79ffb700 1 -- 172.21.15.3:0/3946862751 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6800f6c0 con 0x7f3f7c1051e0 2023-07-20T01:41:59.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.757+0000 7f3f830aa700 1 -- 172.21.15.3:0/3946862751 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c1a35c0 con 0x7f3f7c1051e0 2023-07-20T01:41:59.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.758+0000 7f3f830aa700 1 -- 172.21.15.3:0/3946862751 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7c04e470 con 0x7f3f7c1051e0 2023-07-20T01:41:59.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.759+0000 7f3f79ffb700 1 -- 172.21.15.3:0/3946862751 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3f680225e0 con 0x7f3f7c1051e0 2023-07-20T01:41:59.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.759+0000 7f3f79ffb700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3f6c0a1150 0x7f3f6c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:41:59.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.760+0000 7f3f80e46700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3f6c0a1150 0x7f3f6c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:41:59.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.760+0000 7f3f79ffb700 1 -- 172.21.15.3:0/3946862751 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3f680ce0a0 con 0x7f3f7c1051e0 2023-07-20T01:41:59.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.760+0000 7f3f80e46700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3f6c0a1150 0x7f3f6c0a3610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f70005fd0 tx=0x7f3f7000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:41:59.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.763+0000 7f3f79ffb700 1 -- 172.21.15.3:0/3946862751 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3f68097d20 con 0x7f3f7c1051e0 2023-07-20T01:41:59.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.960+0000 7f3f830aa700 1 -- 172.21.15.3:0/3946862751 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f7c197b60 con 0x7f3f7c1051e0 2023-07-20T01:41:59.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.961+0000 7f3f79ffb700 1 -- 172.21.15.3:0/3946862751 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f6809b340 con 0x7f3f7c1051e0 2023-07-20T01:41:59.929 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-07-20T01:41:59.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 -- 172.21.15.3:0/3946862751 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3f6c0a1150 msgr2=0x7f3f6c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:59.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3f6c0a1150 0x7f3f6c0a3610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f70005fd0 tx=0x7f3f7000a560 comp rx=0 tx=0).stop 2023-07-20T01:41:59.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 -- 172.21.15.3:0/3946862751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c1051e0 msgr2=0x7f3f7c19cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:41:59.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c1051e0 0x7f3f7c19cb10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3f68005910 tx=0x7f3f68003970 comp rx=0 tx=0).stop 2023-07-20T01:41:59.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 -- 172.21.15.3:0/3946862751 shutdown_connections 2023-07-20T01:41:59.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f7c105ba0 0x7f3f7c196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:59.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3f6c0a1150 0x7f3f6c0a3610 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:59.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f7c068730 0x7f3f7c19c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:59.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 --2- 172.21.15.3:0/3946862751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c1051e0 0x7f3f7c19cb10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:41:59.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 -- 172.21.15.3:0/3946862751 >> 172.21.15.3:0/3946862751 conn(0x7f3f7c076770 msgr2=0x7f3f7c0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:41:59.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 -- 172.21.15.3:0/3946862751 shutdown_connections 2023-07-20T01:41:59.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:41:59.963+0000 7f3f830aa700 1 -- 172.21.15.3:0/3946862751 wait complete. 2023-07-20T01:42:00.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:41:59 smithi003 ceph-mon[96422]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:00.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:41:59 smithi043 ceph-mon[97823]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:00.262 INFO:tasks.cephadm:Adding mds.a on smithi003 2023-07-20T01:42:00.262 INFO:tasks.cephadm:Adding mds.d on smithi003 2023-07-20T01:42:00.262 INFO:tasks.cephadm:Adding mds.g on smithi003 2023-07-20T01:42:00.262 INFO:tasks.cephadm:Adding mds.j on smithi003 2023-07-20T01:42:00.263 INFO:tasks.cephadm:Adding mds.b on smithi043 2023-07-20T01:42:00.263 INFO:tasks.cephadm:Adding mds.e on smithi043 2023-07-20T01:42:00.263 INFO:tasks.cephadm:Adding mds.h on smithi043 2023-07-20T01:42:00.263 INFO:tasks.cephadm:Adding mds.k on smithi043 2023-07-20T01:42:00.263 INFO:tasks.cephadm:Adding mds.c on smithi086 2023-07-20T01:42:00.264 INFO:tasks.cephadm:Adding mds.f on smithi086 2023-07-20T01:42:00.264 INFO:tasks.cephadm:Adding mds.i on smithi086 2023-07-20T01:42:00.264 INFO:tasks.cephadm:Adding mds.l on smithi086 2023-07-20T01:42:00.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c -- ceph orch apply mds all '12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi043=b;smithi043=e;smithi043=h;smithi043=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l' 2023-07-20T01:42:00.300 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:41:59 smithi086 ceph-mon[98138]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:00.834 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:42:01.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:00 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3946862751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:42:01.212 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:00 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3946862751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:42:01.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:00 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3946862751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:42:01.988 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.020+0000 7f2a3cc35700 1 -- 172.21.15.86:0/2431369735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a38100800 msgr2=0x7f2a38100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:01.988 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.020+0000 7f2a3cc35700 1 --2- 172.21.15.86:0/2431369735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a38100800 0x7f2a38100c20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009990 tx=0x7f2a2c009ca0 comp rx=0 tx=0).stop 2023-07-20T01:42:01.988 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.020+0000 7f2a3cc35700 1 -- 172.21.15.86:0/2431369735 shutdown_connections 2023-07-20T01:42:01.989 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.020+0000 7f2a3cc35700 1 --2- 172.21.15.86:0/2431369735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a381023a0 0x7f2a38108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:01.989 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.020+0000 7f2a3cc35700 1 --2- 172.21.15.86:0/2431369735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a38100800 0x7f2a38100c20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:01.989 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.020+0000 7f2a3cc35700 1 --2- 172.21.15.86:0/2431369735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a38101a00 0x7f2a38101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:01.989 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.020+0000 7f2a3cc35700 1 -- 172.21.15.86:0/2431369735 >> 172.21.15.86:0/2431369735 conn(0x7f2a380fbd60 msgr2=0x7f2a380fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:01.989 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.020+0000 7f2a3cc35700 1 -- 172.21.15.86:0/2431369735 shutdown_connections 2023-07-20T01:42:01.989 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.020+0000 7f2a3cc35700 1 -- 172.21.15.86:0/2431369735 wait complete. 2023-07-20T01:42:01.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.021+0000 7f2a3cc35700 1 Processor -- start 2023-07-20T01:42:01.991 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.021+0000 7f2a3cc35700 1 -- start start 2023-07-20T01:42:01.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.021+0000 7f2a3cc35700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a38100800 0x7f2a38074510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:01.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.021+0000 7f2a3cc35700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a38101a00 0x7f2a38072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:01.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.021+0000 7f2a3cc35700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a381023a0 0x7f2a380730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:01.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.021+0000 7f2a3cc35700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a38073820 con 0x7f2a381023a0 2023-07-20T01:42:01.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.021+0000 7f2a3cc35700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a38073990 con 0x7f2a38101a00 2023-07-20T01:42:01.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.021+0000 7f2a3cc35700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a381a0590 con 0x7f2a38100800 2023-07-20T01:42:01.992 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.021+0000 7f2a3659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a38100800 0x7f2a38074510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:01.993 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.021+0000 7f2a3659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a38100800 0x7f2a38074510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58906/0 (socket says 172.21.15.86:58906) 2023-07-20T01:42:01.993 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.021+0000 7f2a3659c700 1 -- 172.21.15.86:0/3175520388 learned_addr learned my addr 172.21.15.86:0/3175520388 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-07-20T01:42:01.993 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a35d9b700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a38101a00 0x7f2a38072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:01.993 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a36d9d700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a381023a0 0x7f2a380730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:01.993 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a3659c700 1 -- 172.21.15.86:0/3175520388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a38101a00 msgr2=0x7f2a38072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:01.993 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a3659c700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a38101a00 0x7f2a38072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:01.993 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a3659c700 1 -- 172.21.15.86:0/3175520388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a381023a0 msgr2=0x7f2a380730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:01.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a3659c700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a381023a0 0x7f2a380730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:01.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a3659c700 1 -- 172.21.15.86:0/3175520388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2c0095f0 con 0x7f2a38100800 2023-07-20T01:42:01.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a3659c700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a38100800 0x7f2a38074510 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c005b40 tx=0x7f2a2c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:01.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a277fe700 1 -- 172.21.15.86:0/3175520388 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c01d070 con 0x7f2a38100800 2023-07-20T01:42:01.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a277fe700 1 -- 172.21.15.86:0/3175520388 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a2c003e20 con 0x7f2a38100800 2023-07-20T01:42:01.994 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a3cc35700 1 -- 172.21.15.86:0/3175520388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a381a0810 con 0x7f2a38100800 2023-07-20T01:42:01.996 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a277fe700 1 -- 172.21.15.86:0/3175520388 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c005630 con 0x7f2a38100800 2023-07-20T01:42:01.996 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.022+0000 7f2a3cc35700 1 -- 172.21.15.86:0/3175520388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a381a0c20 con 0x7f2a38100800 2023-07-20T01:42:01.996 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.023+0000 7f2a3cc35700 1 -- 172.21.15.86:0/3175520388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a38107e40 con 0x7f2a38100800 2023-07-20T01:42:01.997 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.024+0000 7f2a277fe700 1 -- 172.21.15.86:0/3175520388 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f2a2c00fc80 con 0x7f2a38100800 2023-07-20T01:42:01.997 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.024+0000 7f2a277fe700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2a200a1150 0x7f2a200a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:01.997 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.025+0000 7f2a277fe700 1 -- 172.21.15.86:0/3175520388 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2a2c0cd680 con 0x7f2a38100800 2023-07-20T01:42:01.997 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.025+0000 7f2a35d9b700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2a200a1150 0x7f2a200a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:01.998 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.025+0000 7f2a35d9b700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2a200a1150 0x7f2a200a3610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2a28005950 tx=0x7f2a2800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:01.998 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.029+0000 7f2a277fe700 1 -- 172.21.15.86:0/3175520388 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2a2c097300 con 0x7f2a38100800 2023-07-20T01:42:02.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:01 smithi086 ceph-mon[98138]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:02.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:01 smithi003 ceph-mon[96422]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:02.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.222+0000 7f2a3cc35700 1 -- 172.21.15.86:0/3175520388 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi043=b;smithi043=e;smithi043=h;smithi043=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a38060e70 con 0x7f2a200a1150 2023-07-20T01:42:02.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.231+0000 7f2a277fe700 1 -- 172.21.15.86:0/3175520388 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f2a38060e70 con 0x7f2a200a1150 2023-07-20T01:42:02.199 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mds.all update... 2023-07-20T01:42:02.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 -- 172.21.15.86:0/3175520388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2a200a1150 msgr2=0x7f2a200a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:02.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2a200a1150 0x7f2a200a3610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2a28005950 tx=0x7f2a2800b410 comp rx=0 tx=0).stop 2023-07-20T01:42:02.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 -- 172.21.15.86:0/3175520388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a38100800 msgr2=0x7f2a38074510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:02.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a38100800 0x7f2a38074510 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c005b40 tx=0x7f2a2c00f690 comp rx=0 tx=0).stop 2023-07-20T01:42:02.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 -- 172.21.15.86:0/3175520388 shutdown_connections 2023-07-20T01:42:02.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a381023a0 0x7f2a380730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:02.202 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a38100800 0x7f2a38074510 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:02.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2a200a1150 0x7f2a200a3610 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:02.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 --2- 172.21.15.86:0/3175520388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a38101a00 0x7f2a38072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:02.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 -- 172.21.15.86:0/3175520388 >> 172.21.15.86:0/3175520388 conn(0x7f2a380fbd60 msgr2=0x7f2a38104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:02.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 -- 172.21.15.86:0/3175520388 shutdown_connections 2023-07-20T01:42:02.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-07-20T01:42:02.233+0000 7f2a3cc35700 1 -- 172.21.15.86:0/3175520388 wait complete. 2023-07-20T01:42:02.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:01 smithi043 ceph-mon[97823]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:02.576 DEBUG:teuthology.orchestra.run.smithi003:mds.a> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.a.service 2023-07-20T01:42:02.579 DEBUG:teuthology.orchestra.run.smithi003:mds.d> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.d.service 2023-07-20T01:42:02.617 INFO:journalctl@ceph.mds.a.smithi003.stdout:-- Logs begin at Thu 2023-07-20 01:20:47 UTC. -- 2023-07-20T01:42:02.618 DEBUG:teuthology.orchestra.run.smithi003:mds.g> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.g.service 2023-07-20T01:42:02.648 INFO:journalctl@ceph.mds.d.smithi003.stdout:-- Logs begin at Thu 2023-07-20 01:20:47 UTC. -- 2023-07-20T01:42:02.649 DEBUG:teuthology.orchestra.run.smithi003:mds.j> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.j.service 2023-07-20T01:42:02.678 INFO:journalctl@ceph.mds.g.smithi003.stdout:-- Logs begin at Thu 2023-07-20 01:20:47 UTC. -- 2023-07-20T01:42:02.679 DEBUG:teuthology.orchestra.run.smithi043:mds.b> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.b.service 2023-07-20T01:42:02.682 DEBUG:teuthology.orchestra.run.smithi043:mds.e> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.e.service 2023-07-20T01:42:02.706 INFO:journalctl@ceph.mds.j.smithi003.stdout:-- Logs begin at Thu 2023-07-20 01:20:47 UTC. -- 2023-07-20T01:42:02.721 INFO:journalctl@ceph.mds.b.smithi043.stdout:-- Logs begin at Thu 2023-07-20 01:20:44 UTC. -- 2023-07-20T01:42:02.722 DEBUG:teuthology.orchestra.run.smithi043:mds.h> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.h.service 2023-07-20T01:42:02.752 INFO:journalctl@ceph.mds.e.smithi043.stdout:-- Logs begin at Thu 2023-07-20 01:20:44 UTC. -- 2023-07-20T01:42:02.753 DEBUG:teuthology.orchestra.run.smithi043:mds.k> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.k.service 2023-07-20T01:42:02.781 INFO:journalctl@ceph.mds.h.smithi043.stdout:-- Logs begin at Thu 2023-07-20 01:20:44 UTC. -- 2023-07-20T01:42:02.782 DEBUG:teuthology.orchestra.run.smithi086:mds.c> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.c.service 2023-07-20T01:42:02.784 DEBUG:teuthology.orchestra.run.smithi086:mds.f> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.f.service 2023-07-20T01:42:02.810 INFO:journalctl@ceph.mds.k.smithi043.stdout:-- Logs begin at Thu 2023-07-20 01:20:44 UTC. -- 2023-07-20T01:42:02.813 INFO:journalctl@ceph.mds.c.smithi086.stdout:-- Logs begin at Thu 2023-07-20 01:20:49 UTC. -- 2023-07-20T01:42:02.814 DEBUG:teuthology.orchestra.run.smithi086:mds.i> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.i.service 2023-07-20T01:42:02.841 INFO:journalctl@ceph.mds.f.smithi086.stdout:-- Logs begin at Thu 2023-07-20 01:20:49 UTC. -- 2023-07-20T01:42:02.842 DEBUG:teuthology.orchestra.run.smithi086:mds.l> sudo journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.l.service 2023-07-20T01:42:02.868 INFO:journalctl@ceph.mds.i.smithi086.stdout:-- Logs begin at Thu 2023-07-20 01:20:49 UTC. -- 2023-07-20T01:42:02.869 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-20T01:42:02.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T01:42:02.896 INFO:journalctl@ceph.mds.l.smithi086.stdout:-- Logs begin at Thu 2023-07-20 01:20:49 UTC. -- 2023-07-20T01:42:03.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.053+0000 7f3ac11a5700 1 -- 172.21.15.3:0/3193239372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3abc0ff760 msgr2=0x7f3abc0ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.053+0000 7f3ac11a5700 1 --2- 172.21.15.3:0/3193239372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3abc0ff760 0x7f3abc0ffb80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0099c0 tx=0x7f3aac009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.053+0000 7f3ac11a5700 1 -- 172.21.15.3:0/3193239372 shutdown_connections 2023-07-20T01:42:03.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.053+0000 7f3ac11a5700 1 --2- 172.21.15.3:0/3193239372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3abc1012f0 0x7f3abc107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.053+0000 7f3ac11a5700 1 --2- 172.21.15.3:0/3193239372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3abc0ff760 0x7f3abc0ffb80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.053+0000 7f3ac11a5700 1 --2- 172.21.15.3:0/3193239372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3abc100950 0x7f3abc100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.053+0000 7f3ac11a5700 1 -- 172.21.15.3:0/3193239372 >> 172.21.15.3:0/3193239372 conn(0x7f3abc0facd0 msgr2=0x7f3abc0fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:03.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.053+0000 7f3ac11a5700 1 -- 172.21.15.3:0/3193239372 shutdown_connections 2023-07-20T01:42:03.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.053+0000 7f3ac11a5700 1 -- 172.21.15.3:0/3193239372 wait complete. 2023-07-20T01:42:03.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.054+0000 7f3ac11a5700 1 Processor -- start 2023-07-20T01:42:03.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.054+0000 7f3ac11a5700 1 -- start start 2023-07-20T01:42:03.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.054+0000 7f3ac11a5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3abc0ff760 0x7f3abc1965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.054+0000 7f3ac11a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3abc100950 0x7f3abc196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.054+0000 7f3ac11a5700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3abc1012f0 0x7f3abc19db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.054+0000 7f3ac11a5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc19e130 con 0x7f3abc0ff760 2023-07-20T01:42:03.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3ac11a5700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc19e270 con 0x7f3abc1012f0 2023-07-20T01:42:03.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3ac11a5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc19e400 con 0x7f3abc100950 2023-07-20T01:42:03.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3abad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3abc0ff760 0x7f3abc1965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3aba59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3abc100950 0x7f3abc196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3abad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3abc0ff760 0x7f3abc1965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42434/0 (socket says 172.21.15.3:42434) 2023-07-20T01:42:03.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3aba59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3abc100950 0x7f3abc196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.3:46048/0 (socket says 172.21.15.3:46048) 2023-07-20T01:42:03.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3abad9d700 1 -- 172.21.15.3:0/38458084 learned_addr learned my addr 172.21.15.3:0/38458084 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:03.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3abb59e700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3abc1012f0 0x7f3abc19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3abad9d700 1 -- 172.21.15.3:0/38458084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3abc100950 msgr2=0x7f3abc196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3abad9d700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3abc100950 0x7f3abc196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3abad9d700 1 -- 172.21.15.3:0/38458084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3abc1012f0 msgr2=0x7f3abc19db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3abad9d700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3abc1012f0 0x7f3abc19db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3abad9d700 1 -- 172.21.15.3:0/38458084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac0095f0 con 0x7f3abc0ff760 2023-07-20T01:42:03.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3aba59c700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3abc100950 0x7f3abc196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:42:03.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3abad9d700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3abc0ff760 0x7f3abc1965b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009990 tx=0x7f3aac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:03.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3ab3fff700 1 -- 172.21.15.3:0/38458084 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac01d070 con 0x7f3abc0ff760 2023-07-20T01:42:03.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3ab3fff700 1 -- 172.21.15.3:0/38458084 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3aac004010 con 0x7f3abc0ff760 2023-07-20T01:42:03.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3ac11a5700 1 -- 172.21.15.3:0/38458084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3abc19e680 con 0x7f3abc0ff760 2023-07-20T01:42:03.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3ab3fff700 1 -- 172.21.15.3:0/38458084 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac022cd0 con 0x7f3abc0ff760 2023-07-20T01:42:03.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.055+0000 7f3ac11a5700 1 -- 172.21.15.3:0/38458084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3abc19eb30 con 0x7f3abc0ff760 2023-07-20T01:42:03.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.056+0000 7f3ac11a5700 1 -- 172.21.15.3:0/38458084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3abc106da0 con 0x7f3abc0ff760 2023-07-20T01:42:03.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.057+0000 7f3ab3fff700 1 -- 172.21.15.3:0/38458084 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3aac022470 con 0x7f3abc0ff760 2023-07-20T01:42:03.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.058+0000 7f3ab3fff700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3aa80a1150 0x7f3aa80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.058+0000 7f3aba59c700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3aa80a1150 0x7f3aa80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.058+0000 7f3ab3fff700 1 -- 172.21.15.3:0/38458084 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3aac0cdab0 con 0x7f3abc0ff760 2023-07-20T01:42:03.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.058+0000 7f3aba59c700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3aa80a1150 0x7f3aa80a3610 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4005950 tx=0x7f3aa400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:03.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.061+0000 7f3ab3fff700 1 -- 172.21.15.3:0/38458084 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3aac097730 con 0x7f3abc0ff760 2023-07-20T01:42:03.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.263+0000 7f3ac11a5700 1 -- 172.21.15.3:0/38458084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3abc02c970 con 0x7f3abc0ff760 2023-07-20T01:42:03.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.264+0000 7f3ab3fff700 1 -- 172.21.15.3:0/38458084 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18240 (secure 0 0 0) 0x7f3aac09ad50 con 0x7f3abc0ff760 2023-07-20T01:42:03.232 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:03.232 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":76,"fsid":"9b4fd2fc-269d-11ee-9b34-001a4aab830c","created":"2023-07-20T01:35:37.746283+0000","modified":"2023-07-20T01:41:54.557580+0000","last_up_change":"2023-07-20T01:41:53.551492+0000","last_in_change":"2023-07-20T01:41:35.988339+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T01:38:42.316631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0073d11-a48f-415c-babd-5672ef2585b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6803","nonce":1627689053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6805","nonce":1627689053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6809","nonce":1627689053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6807","nonce":1627689053}]},"public_addr":"172.21.15.3:6803/1627689053","cluster_addr":"172.21.15.3:6805/1627689053","heartbeat_back_addr":"172.21.15.3:6809/1627689053","heartbeat_front_addr":"172.21.15.3:6807/1627689053","state":["exists","up"]},{"osd":1,"uuid":"91f60b3d-e08c-4a41-b480-c3d30348eacc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6801","nonce":3632963249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6803","nonce":3632963249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6807","nonce":3632963249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6805","nonce":3632963249}]},"public_addr":"172.21.15.43:6801/3632963249","cluster_addr":"172.21.15.43:6803/3632963249","heartbeat_back_addr":"172.21.15.43:6807/3632963249","heartbeat_front_addr":"172.21.15.43:6805/3632963249","state":["exists","up"]},{"osd":2,"uuid":"11808f4b-c50a-44f4-8581-3fc226653b39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6801","nonce":2141120676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6803","nonce":2141120676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6807","nonce":2141120676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6805","nonce":2141120676}]},"public_addr":"172.21.15.86:6801/2141120676","cluster_addr":"172.21.15.86:6803/2141120676","heartbeat_back_addr":"172.21.15.86:6807/2141120676","heartbeat_front_addr":"172.21.15.86:6805/2141120676","state":["exists","up"]},{"osd":3,"uuid":"45c2d3f6-9f5d-4510-9dad-6d95d044725f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6811","nonce":535233193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6813","nonce":535233193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6817","nonce":535233193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6815","nonce":535233193}]},"public_addr":"172.21.15.3:6811/535233193","cluster_addr":"172.21.15.3:6813/535233193","heartbeat_back_addr":"172.21.15.3:6817/535233193","heartbeat_front_addr":"172.21.15.3:6815/535233193","state":["exists","up"]},{"osd":4,"uuid":"7a35bbae-6f6c-4296-9459-f1b4d2de632f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6809","nonce":47431618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6811","nonce":47431618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6815","nonce":47431618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6813","nonce":47431618}]},"public_addr":"172.21.15.43:6809/47431618","cluster_addr":"172.21.15.43:6811/47431618","heartbeat_back_addr":"172.21.15.43:6815/47431618","heartbeat_front_addr":"172.21.15.43:6813/47431618","state":["exists","up"]},{"osd":5,"uuid":"34cccdff-563b-413c-b56c-08839b1534aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6809","nonce":1573392100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1573392100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1573392100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1573392100}]},"public_addr":"172.21.15.86:6809/1573392100","cluster_addr":"172.21.15.86:6811/1573392100","heartbeat_back_addr":"172.21.15.86:6815/1573392100","heartbeat_front_addr":"172.21.15.86:6813/1573392100","state":["exists","up"]},{"osd":6,"uuid":"e1c8f2b2-aed6-4277-b645-6a68e287ca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6819","nonce":1511361132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6821","nonce":1511361132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6825","nonce":1511361132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6823","nonce":1511361132}]},"public_addr":"172.21.15.3:6819/1511361132","cluster_addr":"172.21.15.3:6821/1511361132","heartbeat_back_addr":"172.21.15.3:6825/1511361132","heartbeat_front_addr":"172.21.15.3:6823/1511361132","state":["exists","up"]},{"osd":7,"uuid":"fe6f3c63-9935-43b0-858f-11fd5f91c9ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6817","nonce":3536213360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6819","nonce":3536213360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6823","nonce":3536213360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6821","nonce":3536213360}]},"public_addr":"172.21.15.43:6817/3536213360","cluster_addr":"172.21.15.43:6819/3536213360","heartbeat_back_addr":"172.21.15.43:6823/3536213360","heartbeat_front_addr":"172.21.15.43:6821/3536213360","state":["exists","up"]},{"osd":8,"uuid":"7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6817","nonce":2823536777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6819","nonce":2823536777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6823","nonce":2823536777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6821","nonce":2823536777}]},"public_addr":"172.21.15.86:6817/2823536777","cluster_addr":"172.21.15.86:6819/2823536777","heartbeat_back_addr":"172.21.15.86:6823/2823536777","heartbeat_front_addr":"172.21.15.86:6821/2823536777","state":["exists","up"]},{"osd":9,"uuid":"d356aa99-85dc-4006-9441-e4f7d23d594f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6827","nonce":272617110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6829","nonce":272617110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6833","nonce":272617110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6831","nonce":272617110}]},"public_addr":"172.21.15.3:6827/272617110","cluster_addr":"172.21.15.3:6829/272617110","heartbeat_back_addr":"172.21.15.3:6833/272617110","heartbeat_front_addr":"172.21.15.3:6831/272617110","state":["exists","up"]},{"osd":10,"uuid":"b8848e6f-3991-4755-8fd3-627c9ebcd823","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6825","nonce":161869649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6827","nonce":161869649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6831","nonce":161869649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6829","nonce":161869649}]},"public_addr":"172.21.15.43:6825/161869649","cluster_addr":"172.21.15.43:6827/161869649","heartbeat_back_addr":"172.21.15.43:6831/161869649","heartbeat_front_addr":"172.21.15.43:6829/161869649","state":["exists","up"]},{"osd":11,"uuid":"e0fb9b4d-8559-48c2-9b41-c61a83ca0a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6825","nonce":3393537360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6827","nonce":3393537360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6831","nonce":3393537360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6829","nonce":3393537360}]},"public_addr":"172.21.15.86:6825/3393537360","cluster_addr":"172.21.15.86:6827/3393537360","heartbeat_back_addr":"172.21.15.86:6831/3393537360","heartbeat_front_addr":"172.21.15.86:6829/3393537360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:37:59.181659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:38:20.326807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:38:40.402049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:02.811605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:23.889505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:44.425791+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:06.236232+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:27.676116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:47.846169+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:09.462761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:31.165679+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:51.723548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/942763937":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6801/596448098":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6800/596448098":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6800/417953684":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/1023086410":"2023-07-21T01:36:35.772246+0000","172.21.15.3:0/3900831138":"2023-07-21T01:36:35.772246+0000","172.21.15.3:0/2570287892":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/286795203":"2023-07-21T01:36:10.337325+0000","172.21.15.3:6801/417953684":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/2757371565":"2023-07-21T01:36:10.337325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T01:42:03.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.266+0000 7f3ac11a5700 1 -- 172.21.15.3:0/38458084 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3aa80a1150 msgr2=0x7f3aa80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.266+0000 7f3ac11a5700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3aa80a1150 0x7f3aa80a3610 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4005950 tx=0x7f3aa400b410 comp rx=0 tx=0).stop 2023-07-20T01:42:03.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.266+0000 7f3ac11a5700 1 -- 172.21.15.3:0/38458084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3abc0ff760 msgr2=0x7f3abc1965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.266+0000 7f3ac11a5700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3abc0ff760 0x7f3abc1965b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009990 tx=0x7f3aac004730 comp rx=0 tx=0).stop 2023-07-20T01:42:03.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.266+0000 7f3ac11a5700 1 -- 172.21.15.3:0/38458084 shutdown_connections 2023-07-20T01:42:03.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.266+0000 7f3ac11a5700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f3aa80a1150 0x7f3aa80a3610 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.266+0000 7f3ac11a5700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3abc1012f0 0x7f3abc19db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.267+0000 7f3ac11a5700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3abc0ff760 0x7f3abc1965b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.267+0000 7f3ac11a5700 1 --2- 172.21.15.3:0/38458084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3abc100950 0x7f3abc196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.267+0000 7f3ac11a5700 1 -- 172.21.15.3:0/38458084 >> 172.21.15.3:0/38458084 conn(0x7f3abc0facd0 msgr2=0x7f3abc103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:03.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.267+0000 7f3ac11a5700 1 -- 172.21.15.3:0/38458084 shutdown_connections 2023-07-20T01:42:03.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.267+0000 7f3ac11a5700 1 -- 172.21.15.3:0/38458084 wait complete. 2023-07-20T01:42:03.247 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T01:38:42.316631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T01:42:03.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T01:42:03.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.408+0000 7ff47e3fd700 1 -- 172.21.15.3:0/4113792315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff478100930 msgr2=0x7ff478100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.408+0000 7ff47e3fd700 1 --2- 172.21.15.3:0/4113792315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff478100930 0x7ff478100d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0099c0 tx=0x7ff46c009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.408+0000 7ff47e3fd700 1 -- 172.21.15.3:0/4113792315 shutdown_connections 2023-07-20T01:42:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.408+0000 7ff47e3fd700 1 --2- 172.21.15.3:0/4113792315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4781012d0 0x7ff478107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.408+0000 7ff47e3fd700 1 --2- 172.21.15.3:0/4113792315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4780ff740 0x7ff4780ffb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.408+0000 7ff47e3fd700 1 --2- 172.21.15.3:0/4113792315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff478100930 0x7ff478100d90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.408+0000 7ff47e3fd700 1 -- 172.21.15.3:0/4113792315 >> 172.21.15.3:0/4113792315 conn(0x7ff4780facf0 msgr2=0x7ff4780fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.408+0000 7ff47e3fd700 1 -- 172.21.15.3:0/4113792315 shutdown_connections 2023-07-20T01:42:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.408+0000 7ff47e3fd700 1 -- 172.21.15.3:0/4113792315 wait complete. 2023-07-20T01:42:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff47e3fd700 1 Processor -- start 2023-07-20T01:42:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff47e3fd700 1 -- start start 2023-07-20T01:42:03.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff47e3fd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4780ff740 0x7ff478196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff47e3fd700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff478100930 0x7ff478196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff47e3fd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4781012d0 0x7ff47819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff47e3fd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47819e090 con 0x7ff4780ff740 2023-07-20T01:42:03.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff47e3fd700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47819e1d0 con 0x7ff478100930 2023-07-20T01:42:03.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff47e3fd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47819e360 con 0x7ff4781012d0 2023-07-20T01:42:03.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff477fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4780ff740 0x7ff478196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff477fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4780ff740 0x7ff478196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42448/0 (socket says 172.21.15.3:42448) 2023-07-20T01:42:03.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff477fff700 1 -- 172.21.15.3:0/2855700924 learned_addr learned my addr 172.21.15.3:0/2855700924 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:03.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff47c99a700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4781012d0 0x7ff47819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.409+0000 7ff4777fe700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff478100930 0x7ff478196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff477fff700 1 -- 172.21.15.3:0/2855700924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4781012d0 msgr2=0x7ff47819dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff477fff700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4781012d0 0x7ff47819dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff477fff700 1 -- 172.21.15.3:0/2855700924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff478100930 msgr2=0x7ff478196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff477fff700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff478100930 0x7ff478196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff477fff700 1 -- 172.21.15.3:0/2855700924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c0095f0 con 0x7ff4780ff740 2023-07-20T01:42:03.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff4777fe700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff478100930 0x7ff478196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:03.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff47c99a700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4781012d0 0x7ff47819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:03.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff477fff700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4780ff740 0x7ff478196510 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff46800e970 tx=0x7ff46800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:03.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff4757fa700 1 -- 172.21.15.3:0/2855700924 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468009790 con 0x7ff4780ff740 2023-07-20T01:42:03.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff4757fa700 1 -- 172.21.15.3:0/2855700924 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4680044e0 con 0x7ff4780ff740 2023-07-20T01:42:03.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff47e3fd700 1 -- 172.21.15.3:0/2855700924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47819e640 con 0x7ff4780ff740 2023-07-20T01:42:03.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff4757fa700 1 -- 172.21.15.3:0/2855700924 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468003e30 con 0x7ff4780ff740 2023-07-20T01:42:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.410+0000 7ff47e3fd700 1 -- 172.21.15.3:0/2855700924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47819eba0 con 0x7ff4780ff740 2023-07-20T01:42:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.411+0000 7ff47e3fd700 1 -- 172.21.15.3:0/2855700924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff478106cd0 con 0x7ff4780ff740 2023-07-20T01:42:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.412+0000 7ff4757fa700 1 -- 172.21.15.3:0/2855700924 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7ff468004000 con 0x7ff4780ff740 2023-07-20T01:42:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.413+0000 7ff4757fa700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff4600a11d0 0x7ff4600a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.413+0000 7ff4777fe700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff4600a11d0 0x7ff4600a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.413+0000 7ff4757fa700 1 -- 172.21.15.3:0/2855700924 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff468014070 con 0x7ff4780ff740 2023-07-20T01:42:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.413+0000 7ff4777fe700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff4600a11d0 0x7ff4600a3690 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00b580 tx=0x7ff46c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.416+0000 7ff4757fa700 1 -- 172.21.15.3:0/2855700924 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff4680963c0 con 0x7ff4780ff740 2023-07-20T01:42:03.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:03 smithi043 ceph-mon[97823]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi043=b;smithi043=e;smithi043=h;smithi043=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:42:03.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:03 smithi043 ceph-mon[97823]: Saving service mds.all spec with placement smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi043=b;smithi043=e;smithi043=h;smithi043=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l;count:12 2023-07-20T01:42:03.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:03.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:42:03.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:03.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:42:03.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:03.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:03.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:03.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:03 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:03.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:03 smithi043 ceph-mon[97823]: Deploying daemon mds.l on smithi086 2023-07-20T01:42:03.505 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:03 smithi086 ceph-mon[98138]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi043=b;smithi043=e;smithi043=h;smithi043=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:42:03.505 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:03 smithi086 ceph-mon[98138]: Saving service mds.all spec with placement smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi043=b;smithi043=e;smithi043=h;smithi043=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l;count:12 2023-07-20T01:42:03.505 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:03.505 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:42:03.505 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:03.505 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:42:03.506 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:03.506 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:03.506 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:03.506 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:03 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:03.506 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:03 smithi086 ceph-mon[98138]: Deploying daemon mds.l on smithi086 2023-07-20T01:42:03.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.616+0000 7ff47e3fd700 1 -- 172.21.15.3:0/2855700924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff47802cb00 con 0x7ff4780ff740 2023-07-20T01:42:03.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.616+0000 7ff4757fa700 1 -- 172.21.15.3:0/2855700924 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff46801f340 con 0x7ff4780ff740 2023-07-20T01:42:03.584 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-07-20T01:42:03.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 -- 172.21.15.3:0/2855700924 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff4600a11d0 msgr2=0x7ff4600a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff4600a11d0 0x7ff4600a3690 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00b580 tx=0x7ff46c01a040 comp rx=0 tx=0).stop 2023-07-20T01:42:03.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 -- 172.21.15.3:0/2855700924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4780ff740 msgr2=0x7ff478196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4780ff740 0x7ff478196510 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff46800e970 tx=0x7ff46800ed30 comp rx=0 tx=0).stop 2023-07-20T01:42:03.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 -- 172.21.15.3:0/2855700924 shutdown_connections 2023-07-20T01:42:03.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4781012d0 0x7ff47819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4780ff740 0x7ff478196510 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7ff4600a11d0 0x7ff4600a3690 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 --2- 172.21.15.3:0/2855700924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff478100930 0x7ff478196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 -- 172.21.15.3:0/2855700924 >> 172.21.15.3:0/2855700924 conn(0x7ff4780facf0 msgr2=0x7ff478103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:03.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 -- 172.21.15.3:0/2855700924 shutdown_connections 2023-07-20T01:42:03.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.619+0000 7ff47e3fd700 1 -- 172.21.15.3:0/2855700924 wait complete. 2023-07-20T01:42:03.597 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:03 smithi003 ceph-mon[96422]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi043=b;smithi043=e;smithi043=h;smithi043=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:42:03.598 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:03 smithi003 ceph-mon[96422]: Saving service mds.all spec with placement smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi043=b;smithi043=e;smithi043=h;smithi043=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l;count:12 2023-07-20T01:42:03.598 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:03.598 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:42:03.598 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:03.598 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:42:03.598 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:03.599 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:03.599 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:03.599 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:03 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:03.599 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:03 smithi003 ceph-mon[96422]: Deploying daemon mds.l on smithi086 2023-07-20T01:42:03.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:03.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.758+0000 7f92212df700 1 -- 172.21.15.3:0/3469705677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c0ff720 msgr2=0x7f921c0ffb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.758+0000 7f92212df700 1 --2- 172.21.15.3:0/3469705677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c0ff720 0x7f921c0ffb40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f92100099c0 tx=0x7f9210009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.758+0000 7f92212df700 1 -- 172.21.15.3:0/3469705677 shutdown_connections 2023-07-20T01:42:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.758+0000 7f92212df700 1 --2- 172.21.15.3:0/3469705677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f921c1012b0 0x7f921c107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.758+0000 7f92212df700 1 --2- 172.21.15.3:0/3469705677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c0ff720 0x7f921c0ffb40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.758+0000 7f92212df700 1 --2- 172.21.15.3:0/3469705677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f921c100910 0x7f921c100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.758+0000 7f92212df700 1 -- 172.21.15.3:0/3469705677 >> 172.21.15.3:0/3469705677 conn(0x7f921c0facb0 msgr2=0x7f921c0fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.758+0000 7f92212df700 1 -- 172.21.15.3:0/3469705677 shutdown_connections 2023-07-20T01:42:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.758+0000 7f92212df700 1 -- 172.21.15.3:0/3469705677 wait complete. 2023-07-20T01:42:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.758+0000 7f92212df700 1 Processor -- start 2023-07-20T01:42:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f92212df700 1 -- start start 2023-07-20T01:42:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f92212df700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c0ff720 0x7f921c196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f92212df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f921c100910 0x7f921c196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f92212df700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f921c1012b0 0x7f921c19db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f92212df700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c19e0d0 con 0x7f921c0ff720 2023-07-20T01:42:03.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f92212df700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c19e210 con 0x7f921c1012b0 2023-07-20T01:42:03.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f92212df700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c19e3a0 con 0x7f921c100910 2023-07-20T01:42:03.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f921affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c0ff720 0x7f921c196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f921affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c0ff720 0x7f921c196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42462/0 (socket says 172.21.15.3:42462) 2023-07-20T01:42:03.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f921affd700 1 -- 172.21.15.3:0/3703775633 learned_addr learned my addr 172.21.15.3:0/3703775633 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f921b7fe700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f921c1012b0 0x7f921c19db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f921a7fc700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f921c100910 0x7f921c196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f921affd700 1 -- 172.21.15.3:0/3703775633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f921c100910 msgr2=0x7f921c196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f921affd700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f921c100910 0x7f921c196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f921affd700 1 -- 172.21.15.3:0/3703775633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f921c1012b0 msgr2=0x7f921c19db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f921affd700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f921c1012b0 0x7f921c19db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f921affd700 1 -- 172.21.15.3:0/3703775633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92100095f0 con 0x7f921c0ff720 2023-07-20T01:42:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f921affd700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c0ff720 0x7f921c196550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9210009990 tx=0x7f921000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.759+0000 7f9203fff700 1 -- 172.21.15.3:0/3703775633 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921001d070 con 0x7f921c0ff720 2023-07-20T01:42:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.760+0000 7f9203fff700 1 -- 172.21.15.3:0/3703775633 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9210005030 con 0x7f921c0ff720 2023-07-20T01:42:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.760+0000 7f92212df700 1 -- 172.21.15.3:0/3703775633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921c19e620 con 0x7f921c0ff720 2023-07-20T01:42:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.760+0000 7f9203fff700 1 -- 172.21.15.3:0/3703775633 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210004b30 con 0x7f921c0ff720 2023-07-20T01:42:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.760+0000 7f92212df700 1 -- 172.21.15.3:0/3703775633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921c19ea40 con 0x7f921c0ff720 2023-07-20T01:42:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.761+0000 7f92212df700 1 -- 172.21.15.3:0/3703775633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921c10af80 con 0x7f921c0ff720 2023-07-20T01:42:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.762+0000 7f9203fff700 1 -- 172.21.15.3:0/3703775633 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f9210004cd0 con 0x7f921c0ff720 2023-07-20T01:42:03.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.762+0000 7f9203fff700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f92040a1150 0x7f92040a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:03.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.762+0000 7f921a7fc700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f92040a1150 0x7f92040a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:03.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.762+0000 7f9203fff700 1 -- 172.21.15.3:0/3703775633 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f92100cd2b0 con 0x7f921c0ff720 2023-07-20T01:42:03.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.763+0000 7f921a7fc700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f92040a1150 0x7f92040a3610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f920c009c60 tx=0x7f920c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:03.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:03.767+0000 7f9203fff700 1 -- 172.21.15.3:0/3703775633 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9210096f30 con 0x7f921c0ff720 2023-07-20T01:42:03.852 INFO:journalctl@ceph.mds.l.smithi086.stdout:Jul 20 01:42:03 smithi086 systemd[1]: Starting Ceph mds.l for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:03.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.012+0000 7f92212df700 1 -- 172.21.15.3:0/3703775633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f921c10b170 con 0x7f921c0ff720 2023-07-20T01:42:03.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.013+0000 7f9203fff700 1 -- 172.21.15.3:0/3703775633 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f921009a550 con 0x7f921c0ff720 2023-07-20T01:42:03.981 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:03.981 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-20T01:42:03.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 -- 172.21.15.3:0/3703775633 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f92040a1150 msgr2=0x7f92040a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f92040a1150 0x7f92040a3610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f920c009c60 tx=0x7f920c009260 comp rx=0 tx=0).stop 2023-07-20T01:42:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 -- 172.21.15.3:0/3703775633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c0ff720 msgr2=0x7f921c196550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c0ff720 0x7f921c196550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9210009990 tx=0x7f921000f690 comp rx=0 tx=0).stop 2023-07-20T01:42:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 -- 172.21.15.3:0/3703775633 shutdown_connections 2023-07-20T01:42:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f92040a1150 0x7f92040a3610 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f921c1012b0 0x7f921c19db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c0ff720 0x7f921c196550 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 --2- 172.21.15.3:0/3703775633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f921c100910 0x7f921c196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:03.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 -- 172.21.15.3:0/3703775633 >> 172.21.15.3:0/3703775633 conn(0x7f921c0facb0 msgr2=0x7f921c067a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:03.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 -- 172.21.15.3:0/3703775633 shutdown_connections 2023-07-20T01:42:03.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:04.015+0000 7f92212df700 1 -- 172.21.15.3:0/3703775633 wait complete. 2023-07-20T01:42:03.984 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 1 2023-07-20T01:42:04.190 INFO:journalctl@ceph.mds.l.smithi086.stdout:Jul 20 01:42:03 smithi086 podman[123412]: 2023-07-20T01:42:04.191 INFO:journalctl@ceph.mds.l.smithi086.stdout:Jul 20 01:42:04 smithi086 bash[123412]: a5ef3ba2d84da86e4224f4a2a86d8243ea04867feeaa54d08f3b99b740b02244 2023-07-20T01:42:04.191 INFO:journalctl@ceph.mds.l.smithi086.stdout:Jul 20 01:42:04 smithi086 ceph-mds[123452]: -- 172.21.15.86:0/272549922 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cb1c4791e0 con 0x55cb1c44ec00 2023-07-20T01:42:04.191 INFO:journalctl@ceph.mds.l.smithi086.stdout:Jul 20 01:42:04 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-l[123427]: starting mds.l at 2023-07-20T01:42:04.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:04 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/38458084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:04.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:04 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2855700924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:42:04.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:04 smithi003 ceph-mon[96422]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:04.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:04 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3703775633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:04.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:04 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.86:6832/2824762692,v1:172.21.15.86:6833/2824762692] up:boot 2023-07-20T01:42:04.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:04 smithi003 ceph-mon[96422]: fsmap 1 up:standby 2023-07-20T01:42:04.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:04 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T01:42:04.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:04 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/38458084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:04.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:04 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2855700924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:42:04.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:04 smithi043 ceph-mon[97823]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:04.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:04 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3703775633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:04.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:04 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.86:6832/2824762692,v1:172.21.15.86:6833/2824762692] up:boot 2023-07-20T01:42:04.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:04 smithi043 ceph-mon[97823]: fsmap 1 up:standby 2023-07-20T01:42:04.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:04 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T01:42:04.602 INFO:journalctl@ceph.mds.l.smithi086.stdout:Jul 20 01:42:04 smithi086 systemd[1]: Started Ceph mds.l for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:04.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:04 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/38458084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:04.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:04 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2855700924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:42:04.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:04 smithi086 ceph-mon[98138]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:04.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:04 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3703775633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:04.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:04 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.86:6832/2824762692,v1:172.21.15.86:6833/2824762692] up:boot 2023-07-20T01:42:04.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:04 smithi086 ceph-mon[98138]: fsmap 1 up:standby 2023-07-20T01:42:04.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:04 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T01:42:05.499 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:05.500 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:05.500 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:05.500 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:05.501 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:05.501 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:05 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:05.501 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:05 smithi043 ceph-mon[97823]: Deploying daemon mds.k on smithi043 2023-07-20T01:42:05.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:05.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:05.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:05.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:05.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:05.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:05 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:05.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:05 smithi086 ceph-mon[98138]: Deploying daemon mds.k on smithi043 2023-07-20T01:42:05.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:05.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:05.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:05.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:05.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:05.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:05 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:05.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:05 smithi003 ceph-mon[96422]: Deploying daemon mds.k on smithi043 2023-07-20T01:42:05.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:06.052 INFO:journalctl@ceph.mds.k.smithi043.stdout:Jul 20 01:42:05 smithi043 systemd[1]: Starting Ceph mds.k for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:06.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.168+0000 7fd727ad7700 1 -- 172.21.15.3:0/112255004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7201012e0 msgr2=0x7fd720107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:06.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.168+0000 7fd727ad7700 1 --2- 172.21.15.3:0/112255004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7201012e0 0x7fd720107b90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd71800b0d0 tx=0x7fd71800b3e0 comp rx=0 tx=0).stop 2023-07-20T01:42:06.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 -- 172.21.15.3:0/112255004 shutdown_connections 2023-07-20T01:42:06.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 --2- 172.21.15.3:0/112255004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7201012e0 0x7fd720107b90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:06.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 --2- 172.21.15.3:0/112255004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7200ff750 0x7fd7200ffb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:06.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 --2- 172.21.15.3:0/112255004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd720100940 0x7fd720100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:06.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 -- 172.21.15.3:0/112255004 >> 172.21.15.3:0/112255004 conn(0x7fd7200faca0 msgr2=0x7fd7200fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:06.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 -- 172.21.15.3:0/112255004 shutdown_connections 2023-07-20T01:42:06.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 -- 172.21.15.3:0/112255004 wait complete. 2023-07-20T01:42:06.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 Processor -- start 2023-07-20T01:42:06.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 -- start start 2023-07-20T01:42:06.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7200ff750 0x7fd7201036e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:06.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd720100940 0x7fd720101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:06.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7201012e0 0x7fd720102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:06.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd720102a00 con 0x7fd720100940 2023-07-20T01:42:06.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd720102b70 con 0x7fd7200ff750 2023-07-20T01:42:06.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.169+0000 7fd727ad7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7201a06c0 con 0x7fd7201012e0 2023-07-20T01:42:06.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd725072700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd720100940 0x7fd720101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:06.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd725072700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd720100940 0x7fd720101d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42478/0 (socket says 172.21.15.3:42478) 2023-07-20T01:42:06.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd725873700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7200ff750 0x7fd7201036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:06.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd725072700 1 -- 172.21.15.3:0/3604757652 learned_addr learned my addr 172.21.15.3:0/3604757652 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:06.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd726074700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7201012e0 0x7fd720102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:06.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd725072700 1 -- 172.21.15.3:0/3604757652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7201012e0 msgr2=0x7fd720102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:06.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd725072700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7201012e0 0x7fd720102260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:06.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd725072700 1 -- 172.21.15.3:0/3604757652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7200ff750 msgr2=0x7fd7201036e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:06.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd725072700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7200ff750 0x7fd7201036e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:06.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd725072700 1 -- 172.21.15.3:0/3604757652 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd718009d00 con 0x7fd720100940 2023-07-20T01:42:06.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd726074700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7201012e0 0x7fd720102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:06.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd725072700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd720100940 0x7fd720101d20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00e9a0 tx=0x7fd71c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:06.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd716ffd700 1 -- 172.21.15.3:0/3604757652 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c009790 con 0x7fd720100940 2023-07-20T01:42:06.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd716ffd700 1 -- 172.21.15.3:0/3604757652 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd71c00cbd0 con 0x7fd720100940 2023-07-20T01:42:06.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd727ad7700 1 -- 172.21.15.3:0/3604757652 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7201a09a0 con 0x7fd720100940 2023-07-20T01:42:06.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd716ffd700 1 -- 172.21.15.3:0/3604757652 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c018680 con 0x7fd720100940 2023-07-20T01:42:06.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.170+0000 7fd727ad7700 1 -- 172.21.15.3:0/3604757652 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7201a0f00 con 0x7fd720100940 2023-07-20T01:42:06.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.171+0000 7fd727ad7700 1 -- 172.21.15.3:0/3604757652 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd72010afc0 con 0x7fd720100940 2023-07-20T01:42:06.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.172+0000 7fd716ffd700 1 -- 172.21.15.3:0/3604757652 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fd71c00cd40 con 0x7fd720100940 2023-07-20T01:42:06.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.173+0000 7fd716ffd700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd70c0a11d0 0x7fd70c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:06.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.173+0000 7fd725873700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd70c0a11d0 0x7fd70c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:06.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.173+0000 7fd716ffd700 1 -- 172.21.15.3:0/3604757652 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd71c014070 con 0x7fd720100940 2023-07-20T01:42:06.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.174+0000 7fd725873700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd70c0a11d0 0x7fd70c0a3690 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd7201007a0 tx=0x7fd710009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:06.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.178+0000 7fd716ffd700 1 -- 172.21.15.3:0/3604757652 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd71c051780 con 0x7fd720100940 2023-07-20T01:42:06.370 INFO:journalctl@ceph.mds.k.smithi043.stdout:Jul 20 01:42:06 smithi043 podman[122069]: 2023-07-20T01:42:06.370 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:06 smithi043 ceph-mon[97823]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:06.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.424+0000 7fd727ad7700 1 -- 172.21.15.3:0/3604757652 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd72010b180 con 0x7fd720100940 2023-07-20T01:42:06.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.425+0000 7fd716ffd700 1 -- 172.21.15.3:0/3604757652 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1257 (secure 0 0 0) 0x7fd71c096230 con 0x7fd720100940 2023-07-20T01:42:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:06.393 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2824762692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2824762692},{"type":"v1","addr":"172.21.15.86:6833","nonce":2824762692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T01:42:06.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 -- 172.21.15.3:0/3604757652 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd70c0a11d0 msgr2=0x7fd70c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:06.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd70c0a11d0 0x7fd70c0a3690 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd7201007a0 tx=0x7fd710009260 comp rx=0 tx=0).stop 2023-07-20T01:42:06.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 -- 172.21.15.3:0/3604757652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd720100940 msgr2=0x7fd720101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:06.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd720100940 0x7fd720101d20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00e9a0 tx=0x7fd71c00ecb0 comp rx=0 tx=0).stop 2023-07-20T01:42:06.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 -- 172.21.15.3:0/3604757652 shutdown_connections 2023-07-20T01:42:06.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7201012e0 0x7fd720102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:06.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fd70c0a11d0 0x7fd70c0a3690 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:06.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7200ff750 0x7fd7201036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:06.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 --2- 172.21.15.3:0/3604757652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd720100940 0x7fd720101d20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:06.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 -- 172.21.15.3:0/3604757652 >> 172.21.15.3:0/3604757652 conn(0x7fd7200faca0 msgr2=0x7fd720068bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:06.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 -- 172.21.15.3:0/3604757652 shutdown_connections 2023-07-20T01:42:06.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:06.427+0000 7fd727ad7700 1 -- 172.21.15.3:0/3604757652 wait complete. 2023-07-20T01:42:06.399 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 2 2023-07-20T01:42:06.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:06 smithi003 ceph-mon[96422]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:06.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:06 smithi086 ceph-mon[98138]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:06.752 INFO:journalctl@ceph.mds.k.smithi043.stdout:Jul 20 01:42:06 smithi043 ceph-mds[122109]: -- 172.21.15.43:0/2542214749 <== mon.2 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5628007d51e0 con 0x5628007aa800 2023-07-20T01:42:06.752 INFO:journalctl@ceph.mds.k.smithi043.stdout:Jul 20 01:42:06 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-k[122085]: starting mds.k at 2023-07-20T01:42:06.752 INFO:journalctl@ceph.mds.k.smithi043.stdout:Jul 20 01:42:06 smithi043 bash[122069]: 04ace75247aa66b3c7b1072cf39fc30db1d716e78349cdde6e7119bfa60ec18c 2023-07-20T01:42:06.753 INFO:journalctl@ceph.mds.k.smithi043.stdout:Jul 20 01:42:06 smithi043 systemd[1]: Started Ceph mds.k for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:07.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:07 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3604757652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:07.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:07.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:07.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:07.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:07.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:07.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:07 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:07.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:07 smithi003 ceph-mon[96422]: Deploying daemon mds.j on smithi003 2023-07-20T01:42:07.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:07 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3604757652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:07.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:07.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:07.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:07.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:07.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:07.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:07 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:07.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:07 smithi043 ceph-mon[97823]: Deploying daemon mds.j on smithi003 2023-07-20T01:42:07.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:07 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3604757652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:07.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:07.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:07.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:07.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:07.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:07.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:07 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:07.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:07 smithi086 ceph-mon[98138]: Deploying daemon mds.j on smithi003 2023-07-20T01:42:08.025 INFO:journalctl@ceph.mds.j.smithi003.stdout:Jul 20 01:42:08 smithi003 systemd[1]: Starting Ceph mds.j for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:08.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:08 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.43:6832/2663512295,v1:172.21.15.43:6833/2663512295] up:boot 2023-07-20T01:42:08.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:08 smithi003 ceph-mon[96422]: fsmap 2 up:standby 2023-07-20T01:42:08.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:08 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T01:42:08.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:08 smithi003 ceph-mon[96422]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:08.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:08.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:08 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.43:6832/2663512295,v1:172.21.15.43:6833/2663512295] up:boot 2023-07-20T01:42:08.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:08 smithi043 ceph-mon[97823]: fsmap 2 up:standby 2023-07-20T01:42:08.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:08 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T01:42:08.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:08 smithi043 ceph-mon[97823]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:08.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.572+0000 7f014d54e700 1 -- 172.21.15.3:0/2943334484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0148103c40 msgr2=0x7f01481040a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:08.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.572+0000 7f014d54e700 1 --2- 172.21.15.3:0/2943334484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0148103c40 0x7f01481040a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f013c0099c0 tx=0x7f013c009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:08.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.572+0000 7f014d54e700 1 -- 172.21.15.3:0/2943334484 shutdown_connections 2023-07-20T01:42:08.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.572+0000 7f014d54e700 1 --2- 172.21.15.3:0/2943334484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01481045e0 0x7f014810ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:08.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.572+0000 7f014d54e700 1 --2- 172.21.15.3:0/2943334484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0148102a50 0x7f0148102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:08.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.573+0000 7f014d54e700 1 --2- 172.21.15.3:0/2943334484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0148103c40 0x7f01481040a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:08.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.573+0000 7f014d54e700 1 -- 172.21.15.3:0/2943334484 >> 172.21.15.3:0/2943334484 conn(0x7f01480fdfc0 msgr2=0x7f0148100400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:08.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.573+0000 7f014d54e700 1 -- 172.21.15.3:0/2943334484 shutdown_connections 2023-07-20T01:42:08.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.573+0000 7f014d54e700 1 -- 172.21.15.3:0/2943334484 wait complete. 2023-07-20T01:42:08.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.573+0000 7f014d54e700 1 Processor -- start 2023-07-20T01:42:08.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.573+0000 7f014d54e700 1 -- start start 2023-07-20T01:42:08.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f014d54e700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0148102a50 0x7f0148198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:08.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f014d54e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0148103c40 0x7f0148198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:08.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f014d54e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01481045e0 0x7f014819fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:08.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f014d54e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01481a0310 con 0x7f01481045e0 2023-07-20T01:42:08.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f014d54e700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01481a0450 con 0x7f0148102a50 2023-07-20T01:42:08.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f014d54e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01481a05e0 con 0x7f0148103c40 2023-07-20T01:42:08.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f01477fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01481045e0 0x7f014819fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:08.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f01477fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01481045e0 0x7f014819fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42492/0 (socket says 172.21.15.3:42492) 2023-07-20T01:42:08.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f01477fe700 1 -- 172.21.15.3:0/1715653319 learned_addr learned my addr 172.21.15.3:0/1715653319 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:08.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f0146ffd700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0148102a50 0x7f0148198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:08.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f01467fc700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0148103c40 0x7f0148198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:08.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f01477fe700 1 -- 172.21.15.3:0/1715653319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0148103c40 msgr2=0x7f0148198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:08.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f01477fe700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0148103c40 0x7f0148198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:08.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f01477fe700 1 -- 172.21.15.3:0/1715653319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0148102a50 msgr2=0x7f0148198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:08.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f01477fe700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0148102a50 0x7f0148198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:08.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f01477fe700 1 -- 172.21.15.3:0/1715653319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013c0095f0 con 0x7f01481045e0 2023-07-20T01:42:08.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f01467fc700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0148103c40 0x7f0148198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:08.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f01477fe700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01481045e0 0x7f014819fd40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f014000ada0 tx=0x7f014000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:08.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f012ffff700 1 -- 172.21.15.3:0/1715653319 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014000f070 con 0x7f01481045e0 2023-07-20T01:42:08.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f014d54e700 1 -- 172.21.15.3:0/1715653319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01481a08c0 con 0x7f01481045e0 2023-07-20T01:42:08.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f012ffff700 1 -- 172.21.15.3:0/1715653319 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0140007b30 con 0x7f01481045e0 2023-07-20T01:42:08.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.574+0000 7f012ffff700 1 -- 172.21.15.3:0/1715653319 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01400053e0 con 0x7f01481045e0 2023-07-20T01:42:08.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.575+0000 7f014d54e700 1 -- 172.21.15.3:0/1715653319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01481a0e20 con 0x7f01481045e0 2023-07-20T01:42:08.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.575+0000 7f014d54e700 1 -- 172.21.15.3:0/1715653319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014810a030 con 0x7f01481045e0 2023-07-20T01:42:08.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.576+0000 7f012ffff700 1 -- 172.21.15.3:0/1715653319 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f014000c980 con 0x7f01481045e0 2023-07-20T01:42:08.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.577+0000 7f012ffff700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01300a1150 0x7f01300a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:08.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.577+0000 7f012ffff700 1 -- 172.21.15.3:0/1715653319 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f01400cba10 con 0x7f01481045e0 2023-07-20T01:42:08.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.577+0000 7f0146ffd700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01300a1150 0x7f01300a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:08.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.578+0000 7f0146ffd700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01300a1150 0x7f01300a3610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0148103aa0 tx=0x7f0138009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:08.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.582+0000 7f012ffff700 1 -- 172.21.15.3:0/1715653319 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0140095690 con 0x7f01481045e0 2023-07-20T01:42:08.552 INFO:journalctl@ceph.mds.j.smithi003.stdout:Jul 20 01:42:08 smithi003 podman[129913]: 2023-07-20T01:42:08.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:08 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.43:6832/2663512295,v1:172.21.15.43:6833/2663512295] up:boot 2023-07-20T01:42:08.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:08 smithi086 ceph-mon[98138]: fsmap 2 up:standby 2023-07-20T01:42:08.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:08 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T01:42:08.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:08 smithi086 ceph-mon[98138]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:08.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.837+0000 7f014d54e700 1 -- 172.21.15.3:0/1715653319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f014802c970 con 0x7f01481045e0 2023-07-20T01:42:08.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.837+0000 7f012ffff700 1 -- 172.21.15.3:0/1715653319 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2014 (secure 0 0 0) 0x7f0140098cb0 con 0x7f01481045e0 2023-07-20T01:42:08.805 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:08.805 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6833/2663512295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2663512295},{"type":"v1","addr":"172.21.15.43:6833","nonce":2663512295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2824762692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2824762692},{"type":"v1","addr":"172.21.15.86:6833","nonce":2824762692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T01:42:08.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 -- 172.21.15.3:0/1715653319 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01300a1150 msgr2=0x7f01300a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:08.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01300a1150 0x7f01300a3610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0148103aa0 tx=0x7f0138009190 comp rx=0 tx=0).stop 2023-07-20T01:42:08.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 -- 172.21.15.3:0/1715653319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01481045e0 msgr2=0x7f014819fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01481045e0 0x7f014819fd40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f014000ada0 tx=0x7f014000c330 comp rx=0 tx=0).stop 2023-07-20T01:42:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 -- 172.21.15.3:0/1715653319 shutdown_connections 2023-07-20T01:42:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01481045e0 0x7f014819fd40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f01300a1150 0x7f01300a3610 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0148102a50 0x7f0148198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 --2- 172.21.15.3:0/1715653319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0148103c40 0x7f0148198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 -- 172.21.15.3:0/1715653319 >> 172.21.15.3:0/1715653319 conn(0x7f01480fdfc0 msgr2=0x7f0148107170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 -- 172.21.15.3:0/1715653319 shutdown_connections 2023-07-20T01:42:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:08.840+0000 7f014d54e700 1 -- 172.21.15.3:0/1715653319 wait complete. 2023-07-20T01:42:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 3 2023-07-20T01:42:08.819 INFO:journalctl@ceph.mds.j.smithi003.stdout:Jul 20 01:42:08 smithi003 bash[129913]: 700781a71d54e89da9b15110ef6ca2e9d02a9e471ee33b57b1a5f1d269d43cfc 2023-07-20T01:42:08.819 INFO:journalctl@ceph.mds.j.smithi003.stdout:Jul 20 01:42:08 smithi003 ceph-mds[129982]: -- 172.21.15.3:0/2015694345 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558d6c59b1e0 con 0x558d6c570800 2023-07-20T01:42:08.819 INFO:journalctl@ceph.mds.j.smithi003.stdout:Jul 20 01:42:08 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-j[129929]: starting mds.j at 2023-07-20T01:42:08.819 INFO:journalctl@ceph.mds.j.smithi003.stdout:Jul 20 01:42:08 smithi003 systemd[1]: Started Ceph mds.j for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:09.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:09 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:09.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:09 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:09.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:09 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:09.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:09 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:09.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:09 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:09.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:09 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:09.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:09 smithi003 ceph-mon[96422]: Deploying daemon mds.d on smithi003 2023-07-20T01:42:09.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:09 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/1715653319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:09.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:09 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.3:6834/3626612235,v1:172.21.15.3:6835/3626612235] up:boot 2023-07-20T01:42:09.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:09 smithi003 ceph-mon[96422]: fsmap 3 up:standby 2023-07-20T01:42:09.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:09 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T01:42:10.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:09 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:10.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:09 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:10.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:09 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:10.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:09 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:10.003 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:09 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:10.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:09 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:10.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:09 smithi043 ceph-mon[97823]: Deploying daemon mds.d on smithi003 2023-07-20T01:42:10.004 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:09 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/1715653319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:10.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:09 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.3:6834/3626612235,v1:172.21.15.3:6835/3626612235] up:boot 2023-07-20T01:42:10.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:09 smithi043 ceph-mon[97823]: fsmap 3 up:standby 2023-07-20T01:42:10.005 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:09 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T01:42:10.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:09 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:10.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:09 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:10.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:09 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:10.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:09 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:10.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:09 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:10.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:09 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:10.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:09 smithi086 ceph-mon[98138]: Deploying daemon mds.d on smithi003 2023-07-20T01:42:10.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:09 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/1715653319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:10.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:09 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.3:6834/3626612235,v1:172.21.15.3:6835/3626612235] up:boot 2023-07-20T01:42:10.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:09 smithi086 ceph-mon[98138]: fsmap 3 up:standby 2023-07-20T01:42:10.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:09 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T01:42:10.436 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:42:10 smithi003 systemd[1]: Starting Ceph mds.d for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:10.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:10 smithi003 ceph-mon[96422]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:10.711 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:42:10 smithi003 podman[130299]: 2023-07-20T01:42:10.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:10.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.988+0000 7efe3b984700 1 -- 172.21.15.3:0/729568038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe34100830 msgr2=0x7efe34100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.988+0000 7efe3b984700 1 --2- 172.21.15.3:0/729568038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe34100830 0x7efe34100c50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efe300099c0 tx=0x7efe30009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.988+0000 7efe3b984700 1 -- 172.21.15.3:0/729568038 shutdown_connections 2023-07-20T01:42:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.988+0000 7efe3b984700 1 --2- 172.21.15.3:0/729568038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe341023c0 0x7efe34108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.988+0000 7efe3b984700 1 --2- 172.21.15.3:0/729568038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe34100830 0x7efe34100c50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.988+0000 7efe3b984700 1 --2- 172.21.15.3:0/729568038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe34101a20 0x7efe34101e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.988+0000 7efe3b984700 1 -- 172.21.15.3:0/729568038 >> 172.21.15.3:0/729568038 conn(0x7efe340fbdc0 msgr2=0x7efe340fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.988+0000 7efe3b984700 1 -- 172.21.15.3:0/729568038 shutdown_connections 2023-07-20T01:42:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.988+0000 7efe3b984700 1 -- 172.21.15.3:0/729568038 wait complete. 2023-07-20T01:42:10.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe3b984700 1 Processor -- start 2023-07-20T01:42:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe3b984700 1 -- start start 2023-07-20T01:42:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe3b984700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe34100830 0x7efe34196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe3b984700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe34101a20 0x7efe34196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe3b984700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe341023c0 0x7efe3419dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe3b984700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3419e080 con 0x7efe34100830 2023-07-20T01:42:10.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe3b984700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3419e1c0 con 0x7efe34101a20 2023-07-20T01:42:10.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe3b984700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3419e350 con 0x7efe341023c0 2023-07-20T01:42:10.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe39720700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe34100830 0x7efe34196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:10.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe39720700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe34100830 0x7efe34196500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42526/0 (socket says 172.21.15.3:42526) 2023-07-20T01:42:10.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe39720700 1 -- 172.21.15.3:0/4042758897 learned_addr learned my addr 172.21.15.3:0/4042758897 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:10.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe38f1f700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe34101a20 0x7efe34196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:10.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe39f21700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe341023c0 0x7efe3419dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:10.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe39720700 1 -- 172.21.15.3:0/4042758897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe341023c0 msgr2=0x7efe3419dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe39720700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe341023c0 0x7efe3419dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe39720700 1 -- 172.21.15.3:0/4042758897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe34101a20 msgr2=0x7efe34196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe39720700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe34101a20 0x7efe34196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe39720700 1 -- 172.21.15.3:0/4042758897 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe300095f0 con 0x7efe34100830 2023-07-20T01:42:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe39f21700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe341023c0 0x7efe3419dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.989+0000 7efe38f1f700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe34101a20 0x7efe34196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.990+0000 7efe39720700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe34100830 0x7efe34196500 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efe30009fd0 tx=0x7efe3000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.990+0000 7efe2a7fc700 1 -- 172.21.15.3:0/4042758897 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3001d070 con 0x7efe34100830 2023-07-20T01:42:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.990+0000 7efe3b984700 1 -- 172.21.15.3:0/4042758897 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe3419e5d0 con 0x7efe34100830 2023-07-20T01:42:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.990+0000 7efe2a7fc700 1 -- 172.21.15.3:0/4042758897 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe300054f0 con 0x7efe34100830 2023-07-20T01:42:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.990+0000 7efe2a7fc700 1 -- 172.21.15.3:0/4042758897 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe30004c40 con 0x7efe34100830 2023-07-20T01:42:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.990+0000 7efe3b984700 1 -- 172.21.15.3:0/4042758897 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe3419ea50 con 0x7efe34100830 2023-07-20T01:42:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.991+0000 7efe3b984700 1 -- 172.21.15.3:0/4042758897 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe3410c040 con 0x7efe34100830 2023-07-20T01:42:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.992+0000 7efe2a7fc700 1 -- 172.21.15.3:0/4042758897 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7efe3000fc30 con 0x7efe34100830 2023-07-20T01:42:10.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.992+0000 7efe2a7fc700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efe200a1150 0x7efe200a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:10.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.993+0000 7efe2a7fc700 1 -- 172.21.15.3:0/4042758897 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efe300cd780 con 0x7efe34100830 2023-07-20T01:42:10.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.993+0000 7efe38f1f700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efe200a1150 0x7efe200a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:10.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.993+0000 7efe38f1f700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efe200a1150 0x7efe200a3610 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efe24005fd0 tx=0x7efe24005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:10.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:10.996+0000 7efe2a7fc700 1 -- 172.21.15.3:0/4042758897 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efe30097400 con 0x7efe34100830 2023-07-20T01:42:11.001 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:10 smithi043 ceph-mon[97823]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:11.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:10 smithi086 ceph-mon[98138]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:11.186 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:42:10 smithi003 bash[130299]: f87ba749e8493c61fe4ad726cb7ac708a67a0f95e14ff36a2c077b92f4a31928 2023-07-20T01:42:11.187 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:42:10 smithi003 ceph-mds[130339]: -- 172.21.15.3:0/4057855278 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55659c4c51e0 con 0x55659c49a800 2023-07-20T01:42:11.187 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:42:10 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-d[130315]: starting mds.d at 2023-07-20T01:42:11.187 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:42:10 smithi003 systemd[1]: Started Ceph mds.d for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:11.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.246+0000 7efe3b984700 1 -- 172.21.15.3:0/4042758897 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe34107760 con 0x7efe34100830 2023-07-20T01:42:11.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.246+0000 7efe2a7fc700 1 -- 172.21.15.3:0/4042758897 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2768 (secure 0 0 0) 0x7efe3009aa20 con 0x7efe34100830 2023-07-20T01:42:11.214 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:11.214 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3626612235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3626612235},{"type":"v1","addr":"172.21.15.3:6835","nonce":3626612235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6833/2663512295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2663512295},{"type":"v1","addr":"172.21.15.43:6833","nonce":2663512295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2824762692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2824762692},{"type":"v1","addr":"172.21.15.86:6833","nonce":2824762692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T01:42:11.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 -- 172.21.15.3:0/4042758897 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efe200a1150 msgr2=0x7efe200a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:11.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efe200a1150 0x7efe200a3610 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efe24005fd0 tx=0x7efe24005f60 comp rx=0 tx=0).stop 2023-07-20T01:42:11.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 -- 172.21.15.3:0/4042758897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe34100830 msgr2=0x7efe34196500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:11.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe34100830 0x7efe34196500 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efe30009fd0 tx=0x7efe3000f740 comp rx=0 tx=0).stop 2023-07-20T01:42:11.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 -- 172.21.15.3:0/4042758897 shutdown_connections 2023-07-20T01:42:11.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe341023c0 0x7efe3419dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:11.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe34100830 0x7efe34196500 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:11.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efe200a1150 0x7efe200a3610 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:11.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 --2- 172.21.15.3:0/4042758897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe34101a20 0x7efe34196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:11.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 -- 172.21.15.3:0/4042758897 >> 172.21.15.3:0/4042758897 conn(0x7efe340fbdc0 msgr2=0x7efe340fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:11.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 -- 172.21.15.3:0/4042758897 shutdown_connections 2023-07-20T01:42:11.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:11.249+0000 7efe3b984700 1 -- 172.21.15.3:0/4042758897 wait complete. 2023-07-20T01:42:11.217 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 4 2023-07-20T01:42:12.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:12.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:12.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:12.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:12.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:12.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:12.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:11 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/4042758897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:12.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:11 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.3:6836/3145567212,v1:172.21.15.3:6837/3145567212] up:boot 2023-07-20T01:42:12.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:11 smithi086 ceph-mon[98138]: fsmap 4 up:standby 2023-07-20T01:42:12.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:11 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T01:42:12.122 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:12.123 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:12.123 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:12.123 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:12.124 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:12.124 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:12.124 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:11 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/4042758897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:12.125 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:11 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.3:6836/3145567212,v1:172.21.15.3:6837/3145567212] up:boot 2023-07-20T01:42:12.125 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:11 smithi043 ceph-mon[97823]: fsmap 4 up:standby 2023-07-20T01:42:12.125 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:11 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T01:42:12.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:12.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:12.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:12.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:12.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:12.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:12.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:11 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/4042758897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:12.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:11 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.3:6836/3145567212,v1:172.21.15.3:6837/3145567212] up:boot 2023-07-20T01:42:12.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:11 smithi003 ceph-mon[96422]: fsmap 4 up:standby 2023-07-20T01:42:12.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:11 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T01:42:12.692 INFO:journalctl@ceph.mds.e.smithi043.stdout:Jul 20 01:42:12 smithi043 systemd[1]: Starting Ceph mds.e for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:12.944 INFO:journalctl@ceph.mds.e.smithi043.stdout:Jul 20 01:42:12 smithi043 podman[122408]: 2023-07-20T01:42:12.944 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:12 smithi043 ceph-mon[97823]: Deploying daemon mds.e on smithi043 2023-07-20T01:42:12.945 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:12 smithi043 ceph-mon[97823]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:13.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:12 smithi086 ceph-mon[98138]: Deploying daemon mds.e on smithi043 2023-07-20T01:42:13.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:12 smithi086 ceph-mon[98138]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:13.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:12 smithi003 ceph-mon[96422]: Deploying daemon mds.e on smithi043 2023-07-20T01:42:13.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:12 smithi003 ceph-mon[96422]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:13.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:13.252 INFO:journalctl@ceph.mds.e.smithi043.stdout:Jul 20 01:42:12 smithi043 ceph-mds[122448]: -- 172.21.15.43:0/2230291798 <== mon.2 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f96b41f1e0 con 0x55f96b3f5000 2023-07-20T01:42:13.252 INFO:journalctl@ceph.mds.e.smithi043.stdout:Jul 20 01:42:12 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-e[122424]: starting mds.e at 2023-07-20T01:42:13.252 INFO:journalctl@ceph.mds.e.smithi043.stdout:Jul 20 01:42:13 smithi043 bash[122408]: a95cd833fe5a0b3476ed6a1442d94cdfbce6bbfb328b42208d1b0a4a4bbeaf63 2023-07-20T01:42:13.253 INFO:journalctl@ceph.mds.e.smithi043.stdout:Jul 20 01:42:13 smithi043 systemd[1]: Started Ceph mds.e for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:13.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.402+0000 7fdee7376700 1 -- 172.21.15.3:0/1478388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee00ff720 msgr2=0x7fdee00ffb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:13.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.402+0000 7fdee7376700 1 --2- 172.21.15.3:0/1478388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee00ff720 0x7fdee00ffb40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdedc0099c0 tx=0x7fdedc009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:13.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.402+0000 7fdee7376700 1 -- 172.21.15.3:0/1478388 shutdown_connections 2023-07-20T01:42:13.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.402+0000 7fdee7376700 1 --2- 172.21.15.3:0/1478388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdee01012b0 0x7fdee0107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:13.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.402+0000 7fdee7376700 1 --2- 172.21.15.3:0/1478388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee00ff720 0x7fdee00ffb40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:13.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.402+0000 7fdee7376700 1 --2- 172.21.15.3:0/1478388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdee0100910 0x7fdee0100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:13.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.402+0000 7fdee7376700 1 -- 172.21.15.3:0/1478388 >> 172.21.15.3:0/1478388 conn(0x7fdee00facb0 msgr2=0x7fdee00fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:13.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.402+0000 7fdee7376700 1 -- 172.21.15.3:0/1478388 shutdown_connections 2023-07-20T01:42:13.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.402+0000 7fdee7376700 1 -- 172.21.15.3:0/1478388 wait complete. 2023-07-20T01:42:13.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee7376700 1 Processor -- start 2023-07-20T01:42:13.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee7376700 1 -- start start 2023-07-20T01:42:13.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee7376700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee00ff720 0x7fdee0196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:13.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee7376700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdee0100910 0x7fdee0196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:13.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee7376700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdee01012b0 0x7fdee019d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:13.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee7376700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee019df90 con 0x7fdee00ff720 2023-07-20T01:42:13.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee7376700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee019e0d0 con 0x7fdee01012b0 2023-07-20T01:42:13.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee7376700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee019e260 con 0x7fdee0100910 2023-07-20T01:42:13.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee5112700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee00ff720 0x7fdee0196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:13.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee5112700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee00ff720 0x7fdee0196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42542/0 (socket says 172.21.15.3:42542) 2023-07-20T01:42:13.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee5112700 1 -- 172.21.15.3:0/779345285 learned_addr learned my addr 172.21.15.3:0/779345285 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:13.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.403+0000 7fdee4911700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdee0100910 0x7fdee0196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:13.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fdee5913700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdee01012b0 0x7fdee019d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:13.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fdee5112700 1 -- 172.21.15.3:0/779345285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdee0100910 msgr2=0x7fdee0196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:13.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fdee5112700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdee0100910 0x7fdee0196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:13.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fdee5112700 1 -- 172.21.15.3:0/779345285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdee01012b0 msgr2=0x7fdee019d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:13.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fdee5112700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdee01012b0 0x7fdee019d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:13.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fdee5112700 1 -- 172.21.15.3:0/779345285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc0095f0 con 0x7fdee00ff720 2023-07-20T01:42:13.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fdee5913700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdee01012b0 0x7fdee019d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:42:13.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fdee5112700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee00ff720 0x7fdee0196410 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009990 tx=0x7fdedc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:13.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fded27fc700 1 -- 172.21.15.3:0/779345285 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc01d070 con 0x7fdee00ff720 2023-07-20T01:42:13.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fded27fc700 1 -- 172.21.15.3:0/779345285 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdedc004640 con 0x7fdee00ff720 2023-07-20T01:42:13.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fdee7376700 1 -- 172.21.15.3:0/779345285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee019e4e0 con 0x7fdee00ff720 2023-07-20T01:42:13.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fded27fc700 1 -- 172.21.15.3:0/779345285 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc00f460 con 0x7fdee00ff720 2023-07-20T01:42:13.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.404+0000 7fdee7376700 1 -- 172.21.15.3:0/779345285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee019e990 con 0x7fdee00ff720 2023-07-20T01:42:13.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.405+0000 7fdee7376700 1 -- 172.21.15.3:0/779345285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee010af30 con 0x7fdee00ff720 2023-07-20T01:42:13.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.406+0000 7fded27fc700 1 -- 172.21.15.3:0/779345285 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fdedc004050 con 0x7fdee00ff720 2023-07-20T01:42:13.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.407+0000 7fded27fc700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdecc0a1150 0x7fdecc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:13.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.407+0000 7fdee4911700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdecc0a1150 0x7fdecc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:13.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.407+0000 7fded27fc700 1 -- 172.21.15.3:0/779345285 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdedc0cdce0 con 0x7fdee00ff720 2023-07-20T01:42:13.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.407+0000 7fdee4911700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdecc0a1150 0x7fdecc0a3610 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fded4005ea0 tx=0x7fded4005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:13.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.410+0000 7fded27fc700 1 -- 172.21.15.3:0/779345285 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdedc097960 con 0x7fdee00ff720 2023-07-20T01:42:13.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.656+0000 7fdee7376700 1 -- 172.21.15.3:0/779345285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdee019efd0 con 0x7fdee00ff720 2023-07-20T01:42:13.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.656+0000 7fded27fc700 1 -- 172.21.15.3:0/779345285 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3522 (secure 0 0 0) 0x7fdedc027de0 con 0x7fdee00ff720 2023-07-20T01:42:13.624 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:13.624 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3626612235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3626612235},{"type":"v1","addr":"172.21.15.3:6835","nonce":3626612235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3145567212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3145567212},{"type":"v1","addr":"172.21.15.3:6837","nonce":3145567212}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6833/2663512295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2663512295},{"type":"v1","addr":"172.21.15.43:6833","nonce":2663512295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2824762692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2824762692},{"type":"v1","addr":"172.21.15.86:6833","nonce":2824762692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T01:42:13.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 -- 172.21.15.3:0/779345285 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdecc0a1150 msgr2=0x7fdecc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:13.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdecc0a1150 0x7fdecc0a3610 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fded4005ea0 tx=0x7fded4005e30 comp rx=0 tx=0).stop 2023-07-20T01:42:13.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 -- 172.21.15.3:0/779345285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee00ff720 msgr2=0x7fdee0196410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:13.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee00ff720 0x7fdee0196410 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009990 tx=0x7fdedc003ba0 comp rx=0 tx=0).stop 2023-07-20T01:42:13.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 -- 172.21.15.3:0/779345285 shutdown_connections 2023-07-20T01:42:13.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdecc0a1150 0x7fdecc0a3610 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:13.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdee01012b0 0x7fdee019d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdee00ff720 0x7fdee0196410 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 --2- 172.21.15.3:0/779345285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdee0100910 0x7fdee0196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 -- 172.21.15.3:0/779345285 >> 172.21.15.3:0/779345285 conn(0x7fdee00facb0 msgr2=0x7fdee0067a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 -- 172.21.15.3:0/779345285 shutdown_connections 2023-07-20T01:42:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:13.659+0000 7fdee7376700 1 -- 172.21.15.3:0/779345285 wait complete. 2023-07-20T01:42:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 5 2023-07-20T01:42:14.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:14.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:14.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:14.341 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:14.341 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:14.341 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:14.341 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: Deploying daemon mds.f on smithi086 2023-07-20T01:42:14.342 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/779345285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:14.342 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:14.342 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.43:6834/2898918243,v1:172.21.15.43:6835/2898918243] up:boot 2023-07-20T01:42:14.343 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: fsmap 5 up:standby 2023-07-20T01:42:14.343 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:14 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T01:42:14.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:14.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:14.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:14.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:14.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:14.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:14.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: Deploying daemon mds.f on smithi086 2023-07-20T01:42:14.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/779345285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:14.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:14.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.43:6834/2898918243,v1:172.21.15.43:6835/2898918243] up:boot 2023-07-20T01:42:14.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: fsmap 5 up:standby 2023-07-20T01:42:14.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:14 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T01:42:14.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:14.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:14.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:14.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:14.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:14.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:14.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: Deploying daemon mds.f on smithi086 2023-07-20T01:42:14.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/779345285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:14.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:14.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.43:6834/2898918243,v1:172.21.15.43:6835/2898918243] up:boot 2023-07-20T01:42:14.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: fsmap 5 up:standby 2023-07-20T01:42:14.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:14 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T01:42:14.602 INFO:journalctl@ceph.mds.f.smithi086.stdout:Jul 20 01:42:14 smithi086 systemd[1]: Starting Ceph mds.f for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:15.102 INFO:journalctl@ceph.mds.f.smithi086.stdout:Jul 20 01:42:14 smithi086 podman[123751]: 2023-07-20T01:42:15.102 INFO:journalctl@ceph.mds.f.smithi086.stdout:Jul 20 01:42:15 smithi086 ceph-mds[123791]: -- 172.21.15.86:0/3940013049 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5567d3bcb1e0 con 0x5567d3ba0c00 2023-07-20T01:42:15.102 INFO:journalctl@ceph.mds.f.smithi086.stdout:Jul 20 01:42:15 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-f[123767]: starting mds.f at 2023-07-20T01:42:15.602 INFO:journalctl@ceph.mds.f.smithi086.stdout:Jul 20 01:42:15 smithi086 bash[123751]: f8445cd675d4fe19607e51ffafddfbdcecaed81ab52d0922a2b9b02b401ef442 2023-07-20T01:42:15.602 INFO:journalctl@ceph.mds.f.smithi086.stdout:Jul 20 01:42:15 smithi086 systemd[1]: Started Ceph mds.f for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:15.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:15.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.812+0000 7f71c46f2700 1 -- 172.21.15.3:0/3596679326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71bc100910 msgr2=0x7f71bc100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:15.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.812+0000 7f71c46f2700 1 --2- 172.21.15.3:0/3596679326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71bc100910 0x7f71bc100d70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f71b00099c0 tx=0x7f71b0009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:15.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 -- 172.21.15.3:0/3596679326 shutdown_connections 2023-07-20T01:42:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 --2- 172.21.15.3:0/3596679326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc1012b0 0x7f71bc107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 --2- 172.21.15.3:0/3596679326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f71bc0ff720 0x7f71bc0ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 --2- 172.21.15.3:0/3596679326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71bc100910 0x7f71bc100d70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 -- 172.21.15.3:0/3596679326 >> 172.21.15.3:0/3596679326 conn(0x7f71bc0facb0 msgr2=0x7f71bc0fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 -- 172.21.15.3:0/3596679326 shutdown_connections 2023-07-20T01:42:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 -- 172.21.15.3:0/3596679326 wait complete. 2023-07-20T01:42:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 Processor -- start 2023-07-20T01:42:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 -- start start 2023-07-20T01:42:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71bc0ff720 0x7f71bc196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:15.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f71bc100910 0x7f71bc1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:15.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc1012b0 0x7f71bc19da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:15.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc19e000 con 0x7f71bc0ff720 2023-07-20T01:42:15.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc19e140 con 0x7f71bc100910 2023-07-20T01:42:15.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c46f2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc19e2d0 con 0x7f71bc1012b0 2023-07-20T01:42:15.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.813+0000 7f71c248e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71bc0ff720 0x7f71bc196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:15.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c248e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71bc0ff720 0x7f71bc196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42554/0 (socket says 172.21.15.3:42554) 2023-07-20T01:42:15.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c248e700 1 -- 172.21.15.3:0/3099322077 learned_addr learned my addr 172.21.15.3:0/3099322077 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:15.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c2c8f700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc1012b0 0x7f71bc19da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:15.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c1c8d700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f71bc100910 0x7f71bc1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:15.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c248e700 1 -- 172.21.15.3:0/3099322077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc1012b0 msgr2=0x7f71bc19da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:15.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c248e700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc1012b0 0x7f71bc19da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:15.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c248e700 1 -- 172.21.15.3:0/3099322077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f71bc100910 msgr2=0x7f71bc1969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:15.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c248e700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f71bc100910 0x7f71bc1969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:15.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c248e700 1 -- 172.21.15.3:0/3099322077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b00095f0 con 0x7f71bc0ff720 2023-07-20T01:42:15.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c2c8f700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc1012b0 0x7f71bc19da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:15.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c248e700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71bc0ff720 0x7f71bc196480 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f71b800d980 tx=0x7f71b800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:15.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71af7fe700 1 -- 172.21.15.3:0/3099322077 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b800b950 con 0x7f71bc0ff720 2023-07-20T01:42:15.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71af7fe700 1 -- 172.21.15.3:0/3099322077 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71b8007620 con 0x7f71bc0ff720 2023-07-20T01:42:15.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c46f2700 1 -- 172.21.15.3:0/3099322077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71bc19e5b0 con 0x7f71bc0ff720 2023-07-20T01:42:15.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71af7fe700 1 -- 172.21.15.3:0/3099322077 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b800f3c0 con 0x7f71bc0ff720 2023-07-20T01:42:15.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.814+0000 7f71c46f2700 1 -- 172.21.15.3:0/3099322077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc19eac0 con 0x7f71bc0ff720 2023-07-20T01:42:15.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.815+0000 7f71c46f2700 1 -- 172.21.15.3:0/3099322077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71bc10af40 con 0x7f71bc0ff720 2023-07-20T01:42:15.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.817+0000 7f71af7fe700 1 -- 172.21.15.3:0/3099322077 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f71b8010640 con 0x7f71bc0ff720 2023-07-20T01:42:15.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.817+0000 7f71af7fe700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f71a80a1150 0x7f71a80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:15.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.817+0000 7f71c1c8d700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f71a80a1150 0x7f71a80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:15.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.817+0000 7f71af7fe700 1 -- 172.21.15.3:0/3099322077 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f71b80ccbc0 con 0x7f71bc0ff720 2023-07-20T01:42:15.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.818+0000 7f71c1c8d700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f71a80a1150 0x7f71a80a3610 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f71b000b580 tx=0x7f71b0005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:15.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:15.821+0000 7f71af7fe700 1 -- 172.21.15.3:0/3099322077 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f71b80967b0 con 0x7f71bc0ff720 2023-07-20T01:42:16.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.068+0000 7f71c46f2700 1 -- 172.21.15.3:0/3099322077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71bc19f100 con 0x7f71bc0ff720 2023-07-20T01:42:16.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.068+0000 7f71af7fe700 1 -- 172.21.15.3:0/3099322077 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5033 (secure 0 0 0) 0x7f71b800ee60 con 0x7f71bc0ff720 2023-07-20T01:42:16.035 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:16.035 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3626612235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3626612235},{"type":"v1","addr":"172.21.15.3:6835","nonce":3626612235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3145567212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3145567212},{"type":"v1","addr":"172.21.15.3:6837","nonce":3145567212}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6833/2663512295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2663512295},{"type":"v1","addr":"172.21.15.43:6833","nonce":2663512295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2824762692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2824762692},{"type":"v1","addr":"172.21.15.86:6833","nonce":2824762692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2898918243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2898918243},{"type":"v1","addr":"172.21.15.43:6835","nonce":2898918243}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/113908943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":113908943},{"type":"v1","addr":"172.21.15.86:6835","nonce":113908943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-07-20T01:42:16.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.070+0000 7f71c46f2700 1 -- 172.21.15.3:0/3099322077 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f71a80a1150 msgr2=0x7f71a80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:16.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.070+0000 7f71c46f2700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f71a80a1150 0x7f71a80a3610 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f71b000b580 tx=0x7f71b0005bc0 comp rx=0 tx=0).stop 2023-07-20T01:42:16.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.070+0000 7f71c46f2700 1 -- 172.21.15.3:0/3099322077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71bc0ff720 msgr2=0x7f71bc196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:16.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.070+0000 7f71c46f2700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71bc0ff720 0x7f71bc196480 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f71b800d980 tx=0x7f71b800dc90 comp rx=0 tx=0).stop 2023-07-20T01:42:16.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.070+0000 7f71c46f2700 1 -- 172.21.15.3:0/3099322077 shutdown_connections 2023-07-20T01:42:16.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.070+0000 7f71c46f2700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc1012b0 0x7f71bc19da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:16.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.070+0000 7f71c46f2700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71bc0ff720 0x7f71bc196480 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:16.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.070+0000 7f71c46f2700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f71a80a1150 0x7f71a80a3610 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:16.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.070+0000 7f71c46f2700 1 --2- 172.21.15.3:0/3099322077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f71bc100910 0x7f71bc1969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:16.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.070+0000 7f71c46f2700 1 -- 172.21.15.3:0/3099322077 >> 172.21.15.3:0/3099322077 conn(0x7f71bc0facb0 msgr2=0x7f71bc067a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:16.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.071+0000 7f71c46f2700 1 -- 172.21.15.3:0/3099322077 shutdown_connections 2023-07-20T01:42:16.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:16.071+0000 7f71c46f2700 1 -- 172.21.15.3:0/3099322077 wait complete. 2023-07-20T01:42:16.039 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 7 2023-07-20T01:42:16.424 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.86:6834/113908943,v1:172.21.15.86:6835/113908943] up:boot 2023-07-20T01:42:16.424 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: fsmap 6 up:standby 2023-07-20T01:42:16.424 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T01:42:16.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:16.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:16.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:16.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:16.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:16.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:16.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: Deploying daemon mds.c on smithi086 2023-07-20T01:42:16.426 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:16.426 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:16 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3099322077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:16.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.86:6834/113908943,v1:172.21.15.86:6835/113908943] up:boot 2023-07-20T01:42:16.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: fsmap 6 up:standby 2023-07-20T01:42:16.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T01:42:16.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:16.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:16.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:16.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:16.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:16.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:16.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: Deploying daemon mds.c on smithi086 2023-07-20T01:42:16.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:16.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:16 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3099322077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:16.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.86:6834/113908943,v1:172.21.15.86:6835/113908943] up:boot 2023-07-20T01:42:16.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: fsmap 6 up:standby 2023-07-20T01:42:16.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T01:42:16.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:16.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:16.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:16.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:16.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:16.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:16.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: Deploying daemon mds.c on smithi086 2023-07-20T01:42:16.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:16.504 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:16 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3099322077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:17.102 INFO:journalctl@ceph.mds.c.smithi086.stdout:Jul 20 01:42:16 smithi086 systemd[1]: Starting Ceph mds.c for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:17.523 INFO:journalctl@ceph.mds.c.smithi086.stdout:Jul 20 01:42:17 smithi086 podman[124092]: 2023-07-20T01:42:17.524 INFO:journalctl@ceph.mds.c.smithi086.stdout:Jul 20 01:42:17 smithi086 ceph-mds[124133]: -- 172.21.15.86:0/1968016218 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5596b379b1e0 con 0x5596b3771000 2023-07-20T01:42:17.524 INFO:journalctl@ceph.mds.c.smithi086.stdout:Jul 20 01:42:17 smithi086 bash[124092]: dd0e52ee9e2695253c6799dca584995e79875e7c58f3e2ba00bfb39028d42ef4 2023-07-20T01:42:17.524 INFO:journalctl@ceph.mds.c.smithi086.stdout:Jul 20 01:42:17 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-c[124108]: starting mds.c at 2023-07-20T01:42:17.852 INFO:journalctl@ceph.mds.c.smithi086.stdout:Jul 20 01:42:17 smithi086 systemd[1]: Started Ceph mds.c for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:18.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:18.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.225+0000 7fddea378700 1 -- 172.21.15.3:0/2271162620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde4100910 msgr2=0x7fdde4100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:18.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.225+0000 7fddea378700 1 --2- 172.21.15.3:0/2271162620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde4100910 0x7fdde4100d70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fddd80099c0 tx=0x7fddd8009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:18.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.225+0000 7fddea378700 1 -- 172.21.15.3:0/2271162620 shutdown_connections 2023-07-20T01:42:18.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.225+0000 7fddea378700 1 --2- 172.21.15.3:0/2271162620 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde41012b0 0x7fdde4107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.225+0000 7fddea378700 1 --2- 172.21.15.3:0/2271162620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdde40ff720 0x7fdde40ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.225+0000 7fddea378700 1 --2- 172.21.15.3:0/2271162620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde4100910 0x7fdde4100d70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.225+0000 7fddea378700 1 -- 172.21.15.3:0/2271162620 >> 172.21.15.3:0/2271162620 conn(0x7fdde40facb0 msgr2=0x7fdde40fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.225+0000 7fddea378700 1 -- 172.21.15.3:0/2271162620 shutdown_connections 2023-07-20T01:42:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.225+0000 7fddea378700 1 -- 172.21.15.3:0/2271162620 wait complete. 2023-07-20T01:42:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fddea378700 1 Processor -- start 2023-07-20T01:42:18.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fddea378700 1 -- start start 2023-07-20T01:42:18.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fddea378700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde40ff720 0x7fdde41964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:18.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fddea378700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde4100910 0x7fdde4196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:18.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fddea378700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdde41012b0 0x7fdde419da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:18.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fddea378700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde419e060 con 0x7fdde4100910 2023-07-20T01:42:18.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fddea378700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde419e1a0 con 0x7fdde41012b0 2023-07-20T01:42:18.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fddea378700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde419e330 con 0x7fdde40ff720 2023-07-20T01:42:18.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fdde37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde4100910 0x7fdde4196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:18.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fdde37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde4100910 0x7fdde4196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42570/0 (socket says 172.21.15.3:42570) 2023-07-20T01:42:18.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fdde37fe700 1 -- 172.21.15.3:0/1570251161 learned_addr learned my addr 172.21.15.3:0/1570251161 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:18.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fdde3fff700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde40ff720 0x7fdde41964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:18.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fdde8915700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdde41012b0 0x7fdde419da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:18.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fdde37fe700 1 -- 172.21.15.3:0/1570251161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde40ff720 msgr2=0x7fdde41964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:18.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fdde37fe700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde40ff720 0x7fdde41964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:18.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.226+0000 7fdde37fe700 1 -- 172.21.15.3:0/1570251161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdde41012b0 msgr2=0x7fdde419da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:18.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.227+0000 7fdde37fe700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdde41012b0 0x7fdde419da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:18.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.227+0000 7fdde37fe700 1 -- 172.21.15.3:0/1570251161 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd80095f0 con 0x7fdde4100910 2023-07-20T01:42:18.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.227+0000 7fdde8915700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdde41012b0 0x7fdde419da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:18.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.227+0000 7fdde37fe700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde4100910 0x7fdde4196a20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c010 tx=0x7fddd8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:18.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.227+0000 7fdde17fa700 1 -- 172.21.15.3:0/1570251161 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd801d070 con 0x7fdde4100910 2023-07-20T01:42:18.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.227+0000 7fdde17fa700 1 -- 172.21.15.3:0/1570251161 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddd80040d0 con 0x7fdde4100910 2023-07-20T01:42:18.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.227+0000 7fddea378700 1 -- 172.21.15.3:0/1570251161 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde419e5b0 con 0x7fdde4100910 2023-07-20T01:42:18.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.227+0000 7fdde17fa700 1 -- 172.21.15.3:0/1570251161 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd8005290 con 0x7fdde4100910 2023-07-20T01:42:18.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.227+0000 7fddea378700 1 -- 172.21.15.3:0/1570251161 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde419e9d0 con 0x7fdde4100910 2023-07-20T01:42:18.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.228+0000 7fddea378700 1 -- 172.21.15.3:0/1570251161 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde4106650 con 0x7fdde4100910 2023-07-20T01:42:18.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.229+0000 7fdde17fa700 1 -- 172.21.15.3:0/1570251161 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fddd8004810 con 0x7fdde4100910 2023-07-20T01:42:18.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.230+0000 7fdde17fa700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fddcc0a1150 0x7fddcc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:18.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.230+0000 7fdde3fff700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fddcc0a1150 0x7fddcc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:18.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.230+0000 7fdde17fa700 1 -- 172.21.15.3:0/1570251161 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fddd80cd430 con 0x7fdde4100910 2023-07-20T01:42:18.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.230+0000 7fdde3fff700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fddcc0a1150 0x7fddcc0a3610 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdde4100770 tx=0x7fddd4007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:18.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.234+0000 7fdde17fa700 1 -- 172.21.15.3:0/1570251161 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fddd80970b0 con 0x7fdde4100910 2023-07-20T01:42:18.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.480+0000 7fddea378700 1 -- 172.21.15.3:0/1570251161 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdde402c970 con 0x7fdde4100910 2023-07-20T01:42:18.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.480+0000 7fdde17fa700 1 -- 172.21.15.3:0/1570251161 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5790 (secure 0 0 0) 0x7fddd809a6d0 con 0x7fdde4100910 2023-07-20T01:42:18.447 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:18.447 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3626612235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3626612235},{"type":"v1","addr":"172.21.15.3:6835","nonce":3626612235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3145567212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3145567212},{"type":"v1","addr":"172.21.15.3:6837","nonce":3145567212}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6833/2663512295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2663512295},{"type":"v1","addr":"172.21.15.43:6833","nonce":2663512295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2824762692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2824762692},{"type":"v1","addr":"172.21.15.86:6833","nonce":2824762692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2898918243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2898918243},{"type":"v1","addr":"172.21.15.43:6835","nonce":2898918243}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/113908943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":113908943},{"type":"v1","addr":"172.21.15.86:6835","nonce":113908943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2620198024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2620198024},{"type":"v1","addr":"172.21.15.86:6837","nonce":2620198024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-20T01:42:18.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 -- 172.21.15.3:0/1570251161 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fddcc0a1150 msgr2=0x7fddcc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:18.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fddcc0a1150 0x7fddcc0a3610 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdde4100770 tx=0x7fddd4007400 comp rx=0 tx=0).stop 2023-07-20T01:42:18.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 -- 172.21.15.3:0/1570251161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde4100910 msgr2=0x7fdde4196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:18.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde4100910 0x7fdde4196a20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c010 tx=0x7fddd8009580 comp rx=0 tx=0).stop 2023-07-20T01:42:18.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 -- 172.21.15.3:0/1570251161 shutdown_connections 2023-07-20T01:42:18.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fddcc0a1150 0x7fddcc0a3610 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:18.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdde41012b0 0x7fdde419da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:18.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde40ff720 0x7fdde41964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:18.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 --2- 172.21.15.3:0/1570251161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde4100910 0x7fdde4196a20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:18.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 -- 172.21.15.3:0/1570251161 >> 172.21.15.3:0/1570251161 conn(0x7fdde40facb0 msgr2=0x7fdde40fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:18.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 -- 172.21.15.3:0/1570251161 shutdown_connections 2023-07-20T01:42:18.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:18.482+0000 7fddea378700 1 -- 172.21.15.3:0/1570251161 wait complete. 2023-07-20T01:42:18.451 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2023-07-20T01:42:18.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:18.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:18.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:18.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:18.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:18.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:18.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: Deploying daemon mds.h on smithi043 2023-07-20T01:42:18.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:18.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.86:6836/2620198024,v1:172.21.15.86:6837/2620198024] up:boot 2023-07-20T01:42:18.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: fsmap 7 up:standby 2023-07-20T01:42:18.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T01:42:18.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:18 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/1570251161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:18.868 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:18.869 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:18.869 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:18.870 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:18.870 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:18.870 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:18.870 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: Deploying daemon mds.h on smithi043 2023-07-20T01:42:18.871 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:18.871 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.86:6836/2620198024,v1:172.21.15.86:6837/2620198024] up:boot 2023-07-20T01:42:18.871 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: fsmap 7 up:standby 2023-07-20T01:42:18.871 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T01:42:18.871 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:18 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/1570251161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:18.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:18.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:18.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:18.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:18.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:18.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:18.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: Deploying daemon mds.h on smithi043 2023-07-20T01:42:18.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:18.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.86:6836/2620198024,v1:172.21.15.86:6837/2620198024] up:boot 2023-07-20T01:42:18.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: fsmap 7 up:standby 2023-07-20T01:42:18.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T01:42:18.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:18 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/1570251161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:19.227 INFO:journalctl@ceph.mds.h.smithi043.stdout:Jul 20 01:42:19 smithi043 systemd[1]: Starting Ceph mds.h for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:19.826 INFO:journalctl@ceph.mds.h.smithi043.stdout:Jul 20 01:42:19 smithi043 podman[122746]: 2023-07-20T01:42:20.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:19 smithi003 ceph-mon[96422]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:20.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:19 smithi043 ceph-mon[97823]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:20.252 INFO:journalctl@ceph.mds.h.smithi043.stdout:Jul 20 01:42:19 smithi043 ceph-mds[122786]: -- 172.21.15.43:0/3650636029 <== mon.2 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556d2ebfb1e0 con 0x556d2ebd0800 2023-07-20T01:42:20.253 INFO:journalctl@ceph.mds.h.smithi043.stdout:Jul 20 01:42:19 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-h[122762]: starting mds.h at 2023-07-20T01:42:20.253 INFO:journalctl@ceph.mds.h.smithi043.stdout:Jul 20 01:42:19 smithi043 bash[122746]: 236407f1b65878d2b813d81760c40450e9db36a68d3bb3b55e7fbe09afe0e7e0 2023-07-20T01:42:20.253 INFO:journalctl@ceph.mds.h.smithi043.stdout:Jul 20 01:42:19 smithi043 systemd[1]: Started Ceph mds.h for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:20.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:19 smithi086 ceph-mon[98138]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:20.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:20.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.636+0000 7f0efbf63700 1 -- 172.21.15.3:0/3867297762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ef4100910 msgr2=0x7f0ef4100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:20.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.636+0000 7f0efbf63700 1 --2- 172.21.15.3:0/3867297762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ef4100910 0x7f0ef4100d70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40099c0 tx=0x7f0ee4009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:20.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.636+0000 7f0efbf63700 1 -- 172.21.15.3:0/3867297762 shutdown_connections 2023-07-20T01:42:20.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.636+0000 7f0efbf63700 1 --2- 172.21.15.3:0/3867297762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ef41012b0 0x7f0ef4107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:20.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.636+0000 7f0efbf63700 1 --2- 172.21.15.3:0/3867297762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef40ff720 0x7f0ef40ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:20.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.636+0000 7f0efbf63700 1 --2- 172.21.15.3:0/3867297762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ef4100910 0x7f0ef4100d70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:20.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.636+0000 7f0efbf63700 1 -- 172.21.15.3:0/3867297762 >> 172.21.15.3:0/3867297762 conn(0x7f0ef40facb0 msgr2=0x7f0ef40fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:20.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.636+0000 7f0efbf63700 1 -- 172.21.15.3:0/3867297762 shutdown_connections 2023-07-20T01:42:20.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.636+0000 7f0efbf63700 1 -- 172.21.15.3:0/3867297762 wait complete. 2023-07-20T01:42:20.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0efbf63700 1 Processor -- start 2023-07-20T01:42:20.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0efbf63700 1 -- start start 2023-07-20T01:42:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0efbf63700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ef40ff720 0x7f0ef41964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0efbf63700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ef4100910 0x7f0ef4196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0efbf63700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef41012b0 0x7f0ef419daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0efbf63700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419e070 con 0x7f0ef40ff720 2023-07-20T01:42:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0efbf63700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419e1b0 con 0x7f0ef41012b0 2023-07-20T01:42:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0efbf63700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419e340 con 0x7f0ef4100910 2023-07-20T01:42:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0ef9cff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ef40ff720 0x7f0ef41964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0ef9cff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ef40ff720 0x7f0ef41964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42582/0 (socket says 172.21.15.3:42582) 2023-07-20T01:42:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0ef9cff700 1 -- 172.21.15.3:0/934796088 learned_addr learned my addr 172.21.15.3:0/934796088 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.637+0000 7f0ef94fe700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ef4100910 0x7f0ef4196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0efa500700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef41012b0 0x7f0ef419daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0ef9cff700 1 -- 172.21.15.3:0/934796088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ef4100910 msgr2=0x7f0ef4196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0ef9cff700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ef4100910 0x7f0ef4196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0ef9cff700 1 -- 172.21.15.3:0/934796088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef41012b0 msgr2=0x7f0ef419daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0ef9cff700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef41012b0 0x7f0ef419daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0ef9cff700 1 -- 172.21.15.3:0/934796088 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee40095f0 con 0x7f0ef40ff720 2023-07-20T01:42:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0efa500700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef41012b0 0x7f0ef419daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:42:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0ef9cff700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ef40ff720 0x7f0ef41964f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000d980 tx=0x7f0ef000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0eeaffd700 1 -- 172.21.15.3:0/934796088 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0004020 con 0x7f0ef40ff720 2023-07-20T01:42:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0eeaffd700 1 -- 172.21.15.3:0/934796088 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ef0009730 con 0x7f0ef40ff720 2023-07-20T01:42:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0efbf63700 1 -- 172.21.15.3:0/934796088 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef419e620 con 0x7f0ef40ff720 2023-07-20T01:42:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0eeaffd700 1 -- 172.21.15.3:0/934796088 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0003a80 con 0x7f0ef40ff720 2023-07-20T01:42:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.638+0000 7f0efbf63700 1 -- 172.21.15.3:0/934796088 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef419eb80 con 0x7f0ef40ff720 2023-07-20T01:42:20.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.639+0000 7f0efbf63700 1 -- 172.21.15.3:0/934796088 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef4107850 con 0x7f0ef40ff720 2023-07-20T01:42:20.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.640+0000 7f0eeaffd700 1 -- 172.21.15.3:0/934796088 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f0ef00098a0 con 0x7f0ef40ff720 2023-07-20T01:42:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.641+0000 7f0eeaffd700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0ee00a11d0 0x7f0ee00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.641+0000 7f0ef94fe700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0ee00a11d0 0x7f0ee00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.641+0000 7f0eeaffd700 1 -- 172.21.15.3:0/934796088 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0ef00cc430 con 0x7f0ef40ff720 2023-07-20T01:42:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.641+0000 7f0ef94fe700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0ee00a11d0 0x7f0ee00a3690 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400b580 tx=0x7f0ee4005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.644+0000 7f0eeaffd700 1 -- 172.21.15.3:0/934796088 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ef00960a0 con 0x7f0ef40ff720 2023-07-20T01:42:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.889+0000 7f0efbf63700 1 -- 172.21.15.3:0/934796088 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ef402c970 con 0x7f0ef40ff720 2023-07-20T01:42:20.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.889+0000 7f0eeaffd700 1 -- 172.21.15.3:0/934796088 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6547 (secure 0 0 0) 0x7f0ef00996c0 con 0x7f0ef40ff720 2023-07-20T01:42:20.856 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:20.856 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3626612235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3626612235},{"type":"v1","addr":"172.21.15.3:6835","nonce":3626612235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3145567212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3145567212},{"type":"v1","addr":"172.21.15.3:6837","nonce":3145567212}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6833/2663512295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2663512295},{"type":"v1","addr":"172.21.15.43:6833","nonce":2663512295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2824762692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2824762692},{"type":"v1","addr":"172.21.15.86:6833","nonce":2824762692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2898918243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2898918243},{"type":"v1","addr":"172.21.15.43:6835","nonce":2898918243}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/113908943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":113908943},{"type":"v1","addr":"172.21.15.86:6835","nonce":113908943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1379813430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1379813430},{"type":"v1","addr":"172.21.15.43:6837","nonce":1379813430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2620198024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2620198024},{"type":"v1","addr":"172.21.15.86:6837","nonce":2620198024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-20T01:42:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.891+0000 7f0efbf63700 1 -- 172.21.15.3:0/934796088 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0ee00a11d0 msgr2=0x7f0ee00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.891+0000 7f0efbf63700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0ee00a11d0 0x7f0ee00a3690 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400b580 tx=0x7f0ee4005bc0 comp rx=0 tx=0).stop 2023-07-20T01:42:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.891+0000 7f0efbf63700 1 -- 172.21.15.3:0/934796088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ef40ff720 msgr2=0x7f0ef41964f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.891+0000 7f0efbf63700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ef40ff720 0x7f0ef41964f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000d980 tx=0x7f0ef000dc90 comp rx=0 tx=0).stop 2023-07-20T01:42:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.892+0000 7f0efbf63700 1 -- 172.21.15.3:0/934796088 shutdown_connections 2023-07-20T01:42:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.892+0000 7f0efbf63700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f0ee00a11d0 0x7f0ee00a3690 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.892+0000 7f0efbf63700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef41012b0 0x7f0ef419daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.892+0000 7f0efbf63700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ef40ff720 0x7f0ef41964f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.892+0000 7f0efbf63700 1 --2- 172.21.15.3:0/934796088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ef4100910 0x7f0ef4196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.892+0000 7f0efbf63700 1 -- 172.21.15.3:0/934796088 >> 172.21.15.3:0/934796088 conn(0x7f0ef40facb0 msgr2=0x7f0ef40fcf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.892+0000 7f0efbf63700 1 -- 172.21.15.3:0/934796088 shutdown_connections 2023-07-20T01:42:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:20.892+0000 7f0efbf63700 1 -- 172.21.15.3:0/934796088 wait complete. 2023-07-20T01:42:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 9 2023-07-20T01:42:21.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:20 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.43:6836/1379813430,v1:172.21.15.43:6837/1379813430] up:boot 2023-07-20T01:42:21.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:20 smithi086 ceph-mon[98138]: fsmap 8 up:standby 2023-07-20T01:42:21.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:20 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T01:42:21.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:20 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:21.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:20 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:21.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:20 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:21.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:20 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:21.009 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:20 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:21.009 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:20 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:21.009 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:20 smithi086 ceph-mon[98138]: Deploying daemon mds.i on smithi086 2023-07-20T01:42:21.009 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:20 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/934796088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:21.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:20 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.43:6836/1379813430,v1:172.21.15.43:6837/1379813430] up:boot 2023-07-20T01:42:21.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:20 smithi003 ceph-mon[96422]: fsmap 8 up:standby 2023-07-20T01:42:21.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:20 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T01:42:21.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:20 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:21.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:20 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:21.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:20 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:21.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:20 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:21.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:20 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:21.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:20 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:21.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:20 smithi003 ceph-mon[96422]: Deploying daemon mds.i on smithi086 2023-07-20T01:42:21.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:20 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/934796088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:21.251 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:20 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.43:6836/1379813430,v1:172.21.15.43:6837/1379813430] up:boot 2023-07-20T01:42:21.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:20 smithi043 ceph-mon[97823]: fsmap 8 up:standby 2023-07-20T01:42:21.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:20 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T01:42:21.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:20 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:21.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:20 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:21.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:20 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:21.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:20 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:21.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:20 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:21.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:20 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:21.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:20 smithi043 ceph-mon[97823]: Deploying daemon mds.i on smithi086 2023-07-20T01:42:21.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:20 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/934796088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:21.602 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:42:21 smithi086 systemd[1]: Starting Ceph mds.i for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:22.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:21 smithi086 ceph-mon[98138]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:22.102 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:42:21 smithi086 podman[124429]: 2023-07-20T01:42:22.102 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:42:22 smithi086 ceph-mds[124469]: -- 172.21.15.86:0/295800067 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555b99a431e0 con 0x555b99a18c00 2023-07-20T01:42:22.102 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:42:22 smithi086 bash[124429]: f5a001c40f4611956f72a5e6082abd07272ade20f8d8f91309e764e4a9a51f83 2023-07-20T01:42:22.103 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:42:22 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-i[124445]: starting mds.i at 2023-07-20T01:42:22.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:21 smithi003 ceph-mon[96422]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:22.251 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:21 smithi043 ceph-mon[97823]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:22.602 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:42:22 smithi086 systemd[1]: Started Ceph mds.i for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:22.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:23.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.038+0000 7fab87291700 1 -- 172.21.15.3:0/3004677992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab800feae0 msgr2=0x7fab800fef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:23.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.038+0000 7fab87291700 1 --2- 172.21.15.3:0/3004677992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab800feae0 0x7fab800fef00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009990 tx=0x7fab7c009ca0 comp rx=0 tx=0).stop 2023-07-20T01:42:23.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.038+0000 7fab87291700 1 -- 172.21.15.3:0/3004677992 shutdown_connections 2023-07-20T01:42:23.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.038+0000 7fab87291700 1 --2- 172.21.15.3:0/3004677992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab80102450 0x7fab80108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:23.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.038+0000 7fab87291700 1 --2- 172.21.15.3:0/3004677992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab800feae0 0x7fab800fef00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:23.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.038+0000 7fab87291700 1 --2- 172.21.15.3:0/3004677992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab80101ab0 0x7fab80101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:23.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.038+0000 7fab87291700 1 -- 172.21.15.3:0/3004677992 >> 172.21.15.3:0/3004677992 conn(0x7fab800fa410 msgr2=0x7fab800fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:23.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.038+0000 7fab87291700 1 -- 172.21.15.3:0/3004677992 shutdown_connections 2023-07-20T01:42:23.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.038+0000 7fab87291700 1 -- 172.21.15.3:0/3004677992 wait complete. 2023-07-20T01:42:23.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab87291700 1 Processor -- start 2023-07-20T01:42:23.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab87291700 1 -- start start 2023-07-20T01:42:23.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab87291700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab800feae0 0x7fab80196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:23.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab87291700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab80101ab0 0x7fab80196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:23.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab87291700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab80102450 0x7fab8019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:23.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab87291700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8019e080 con 0x7fab80102450 2023-07-20T01:42:23.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab87291700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8019e1c0 con 0x7fab80101ab0 2023-07-20T01:42:23.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab87291700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8019e350 con 0x7fab800feae0 2023-07-20T01:42:23.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab8582e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab80102450 0x7fab8019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:23.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab8502d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab800feae0 0x7fab80196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:23.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab8582e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab80102450 0x7fab8019dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42600/0 (socket says 172.21.15.3:42600) 2023-07-20T01:42:23.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.039+0000 7fab8582e700 1 -- 172.21.15.3:0/3658997610 learned_addr learned my addr 172.21.15.3:0/3658997610 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:23.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab8582e700 1 -- 172.21.15.3:0/3658997610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab800feae0 msgr2=0x7fab80196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:23.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab8582e700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab800feae0 0x7fab80196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:23.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab8582e700 1 -- 172.21.15.3:0/3658997610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab80101ab0 msgr2=0x7fab80196a40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-20T01:42:23.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab8582e700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab80101ab0 0x7fab80196a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:23.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab8582e700 1 -- 172.21.15.3:0/3658997610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c0095f0 con 0x7fab80102450 2023-07-20T01:42:23.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab8502d700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab800feae0 0x7fab80196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:42:23.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab8582e700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab80102450 0x7fab8019dab0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fab78007d70 tx=0x7fab7800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:23.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab767fc700 1 -- 172.21.15.3:0/3658997610 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7800ad40 con 0x7fab80102450 2023-07-20T01:42:23.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab767fc700 1 -- 172.21.15.3:0/3658997610 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab780128e0 con 0x7fab80102450 2023-07-20T01:42:23.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab87291700 1 -- 172.21.15.3:0/3658997610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8019e630 con 0x7fab80102450 2023-07-20T01:42:23.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab767fc700 1 -- 172.21.15.3:0/3658997610 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78011740 con 0x7fab80102450 2023-07-20T01:42:23.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.040+0000 7fab87291700 1 -- 172.21.15.3:0/3658997610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8019eb40 con 0x7fab80102450 2023-07-20T01:42:23.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.041+0000 7fab87291700 1 -- 172.21.15.3:0/3658997610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab80107e00 con 0x7fab80102450 2023-07-20T01:42:23.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.042+0000 7fab767fc700 1 -- 172.21.15.3:0/3658997610 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fab78012a50 con 0x7fab80102450 2023-07-20T01:42:23.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.043+0000 7fab767fc700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab6c0a1150 0x7fab6c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:23.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.043+0000 7fab8502d700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab6c0a1150 0x7fab6c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:23.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.043+0000 7fab767fc700 1 -- 172.21.15.3:0/3658997610 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fab780cc670 con 0x7fab80102450 2023-07-20T01:42:23.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.043+0000 7fab8502d700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab6c0a1150 0x7fab6c0a3610 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00b2e0 tx=0x7fab7c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:23.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.047+0000 7fab767fc700 1 -- 172.21.15.3:0/3658997610 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fab78099cf0 con 0x7fab80102450 2023-07-20T01:42:23.150 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:23.150 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:23.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:23.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:23.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:23.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:23.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.305+0000 7fab87291700 1 -- 172.21.15.3:0/3658997610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab8002c970 con 0x7fab80102450 2023-07-20T01:42:23.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.306+0000 7fab767fc700 1 -- 172.21.15.3:0/3658997610 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7306 (secure 0 0 0) 0x7fab78099880 con 0x7fab80102450 2023-07-20T01:42:23.273 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:23.273 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3626612235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3626612235},{"type":"v1","addr":"172.21.15.3:6835","nonce":3626612235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3145567212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3145567212},{"type":"v1","addr":"172.21.15.3:6837","nonce":3145567212}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6833/2663512295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2663512295},{"type":"v1","addr":"172.21.15.43:6833","nonce":2663512295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2824762692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2824762692},{"type":"v1","addr":"172.21.15.86:6833","nonce":2824762692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2898918243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2898918243},{"type":"v1","addr":"172.21.15.43:6835","nonce":2898918243}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/113908943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":113908943},{"type":"v1","addr":"172.21.15.86:6835","nonce":113908943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1379813430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1379813430},{"type":"v1","addr":"172.21.15.43:6837","nonce":1379813430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2620198024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2620198024},{"type":"v1","addr":"172.21.15.86:6837","nonce":2620198024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/1279793907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1279793907},{"type":"v1","addr":"172.21.15.86:6839","nonce":1279793907}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T01:42:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 -- 172.21.15.3:0/3658997610 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab6c0a1150 msgr2=0x7fab6c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab6c0a1150 0x7fab6c0a3610 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00b2e0 tx=0x7fab7c005fb0 comp rx=0 tx=0).stop 2023-07-20T01:42:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 -- 172.21.15.3:0/3658997610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab80102450 msgr2=0x7fab8019dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab80102450 0x7fab8019dab0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fab78007d70 tx=0x7fab7800a340 comp rx=0 tx=0).stop 2023-07-20T01:42:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 -- 172.21.15.3:0/3658997610 shutdown_connections 2023-07-20T01:42:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab80102450 0x7fab8019dab0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:23.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab800feae0 0x7fab80196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:23.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fab6c0a1150 0x7fab6c0a3610 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:23.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 --2- 172.21.15.3:0/3658997610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab80101ab0 0x7fab80196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:23.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 -- 172.21.15.3:0/3658997610 >> 172.21.15.3:0/3658997610 conn(0x7fab800fa410 msgr2=0x7fab80104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:23.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 -- 172.21.15.3:0/3658997610 shutdown_connections 2023-07-20T01:42:23.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:23.308+0000 7fab87291700 1 -- 172.21.15.3:0/3658997610 wait complete. 2023-07-20T01:42:23.277 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 10 2023-07-20T01:42:23.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:23 smithi003 ceph-mon[96422]: Deploying daemon mds.g on smithi003 2023-07-20T01:42:23.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:23 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.86:6838/1279793907,v1:172.21.15.86:6839/1279793907] up:boot 2023-07-20T01:42:23.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:23 smithi003 ceph-mon[96422]: fsmap 9 up:standby 2023-07-20T01:42:23.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:23 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T01:42:23.501 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:23.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:23.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:23.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:23.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:23.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:23.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:23 smithi043 ceph-mon[97823]: Deploying daemon mds.g on smithi003 2023-07-20T01:42:23.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:23 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.86:6838/1279793907,v1:172.21.15.86:6839/1279793907] up:boot 2023-07-20T01:42:23.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:23 smithi043 ceph-mon[97823]: fsmap 9 up:standby 2023-07-20T01:42:23.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:23 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T01:42:23.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:23.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:23.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:23.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:23.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:23.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:23.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:23 smithi086 ceph-mon[98138]: Deploying daemon mds.g on smithi003 2023-07-20T01:42:23.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:23 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.86:6838/1279793907,v1:172.21.15.86:6839/1279793907] up:boot 2023-07-20T01:42:23.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:23 smithi086 ceph-mon[98138]: fsmap 9 up:standby 2023-07-20T01:42:23.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:23 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T01:42:24.138 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:42:23 smithi003 systemd[1]: Starting Ceph mds.g for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:24.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:24 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3658997610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:24.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:24 smithi003 ceph-mon[96422]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:24.420 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:42:24 smithi003 podman[130912]: 2023-07-20T01:42:24.420 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:42:24 smithi003 ceph-mds[130953]: -- 172.21.15.3:0/2307445869 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55580593d1e0 con 0x555805913000 2023-07-20T01:42:24.420 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:42:24 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-g[130929]: starting mds.g at 2023-07-20T01:42:24.421 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:42:24 smithi003 bash[130912]: 139189d5414045d0e431e67fc917b6d9f5eb62791e61ef9017835b3a1ea823d9 2023-07-20T01:42:24.501 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:24 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3658997610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:24.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:24 smithi043 ceph-mon[97823]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:24.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:24 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3658997610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:24.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:24 smithi086 ceph-mon[98138]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:24.686 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:42:24 smithi003 systemd[1]: Started Ceph mds.g for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:25.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:25.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.455+0000 7f044e73f700 1 -- 172.21.15.3:0/2511511154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04481012b0 msgr2=0x7f0448107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:25.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.455+0000 7f044e73f700 1 --2- 172.21.15.3:0/2511511154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04481012b0 0x7f0448107b60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f044000b0d0 tx=0x7f044000b3e0 comp rx=0 tx=0).stop 2023-07-20T01:42:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.455+0000 7f044e73f700 1 -- 172.21.15.3:0/2511511154 shutdown_connections 2023-07-20T01:42:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.455+0000 7f044e73f700 1 --2- 172.21.15.3:0/2511511154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04481012b0 0x7f0448107b60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.455+0000 7f044e73f700 1 --2- 172.21.15.3:0/2511511154 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04480ff720 0x7f04480ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.455+0000 7f044e73f700 1 --2- 172.21.15.3:0/2511511154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0448100910 0x7f0448100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.455+0000 7f044e73f700 1 -- 172.21.15.3:0/2511511154 >> 172.21.15.3:0/2511511154 conn(0x7f04480facb0 msgr2=0x7f04480fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.455+0000 7f044e73f700 1 -- 172.21.15.3:0/2511511154 shutdown_connections 2023-07-20T01:42:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.455+0000 7f044e73f700 1 -- 172.21.15.3:0/2511511154 wait complete. 2023-07-20T01:42:25.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.455+0000 7f044e73f700 1 Processor -- start 2023-07-20T01:42:25.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.455+0000 7f044e73f700 1 -- start start 2023-07-20T01:42:25.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f044e73f700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04480ff720 0x7f04481036b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:25.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f044e73f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0448100910 0x7f0448101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:25.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f044e73f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04481012b0 0x7f0448102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:25.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f044e73f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04481029d0 con 0x7f0448100910 2023-07-20T01:42:25.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f044e73f700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0448102b40 con 0x7f04480ff720 2023-07-20T01:42:25.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f044e73f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04481a0630 con 0x7f04481012b0 2023-07-20T01:42:25.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04477fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0448100910 0x7f0448101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:25.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04477fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0448100910 0x7f0448101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42624/0 (socket says 172.21.15.3:42624) 2023-07-20T01:42:25.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04477fe700 1 -- 172.21.15.3:0/817138760 learned_addr learned my addr 172.21.15.3:0/817138760 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:25.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f044ccdc700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04481012b0 0x7f0448102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:25.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f0447fff700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04480ff720 0x7f04481036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:25.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04477fe700 1 -- 172.21.15.3:0/817138760 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04481012b0 msgr2=0x7f0448102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:25.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04477fe700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04481012b0 0x7f0448102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:25.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04477fe700 1 -- 172.21.15.3:0/817138760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04480ff720 msgr2=0x7f04481036b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:25.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04477fe700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04480ff720 0x7f04481036b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:25.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04477fe700 1 -- 172.21.15.3:0/817138760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0440009d00 con 0x7f0448100910 2023-07-20T01:42:25.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f044ccdc700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04481012b0 0x7f0448102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:25.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f0447fff700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04480ff720 0x7f04481036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:42:25.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04477fe700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0448100910 0x7f0448101cf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f043c00d950 tx=0x7f043c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:25.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04457fa700 1 -- 172.21.15.3:0/817138760 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c009b40 con 0x7f0448100910 2023-07-20T01:42:25.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04457fa700 1 -- 172.21.15.3:0/817138760 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f043c009ca0 con 0x7f0448100910 2023-07-20T01:42:25.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f044e73f700 1 -- 172.21.15.3:0/817138760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04481a0910 con 0x7f0448100910 2023-07-20T01:42:25.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f04457fa700 1 -- 172.21.15.3:0/817138760 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c00f620 con 0x7f0448100910 2023-07-20T01:42:25.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.456+0000 7f044e73f700 1 -- 172.21.15.3:0/817138760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04481a0e20 con 0x7f0448100910 2023-07-20T01:42:25.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.457+0000 7f044e73f700 1 -- 172.21.15.3:0/817138760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044810af30 con 0x7f0448100910 2023-07-20T01:42:25.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.459+0000 7f04457fa700 1 -- 172.21.15.3:0/817138760 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f043c010460 con 0x7f0448100910 2023-07-20T01:42:25.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.459+0000 7f04457fa700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f04300a1150 0x7f04300a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:25.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.459+0000 7f04457fa700 1 -- 172.21.15.3:0/817138760 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f043c0cc7f0 con 0x7f0448100910 2023-07-20T01:42:25.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.459+0000 7f0447fff700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f04300a1150 0x7f04300a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:25.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.460+0000 7f0447fff700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f04300a1150 0x7f04300a3610 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0438005fd0 tx=0x7f0438005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:25.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.463+0000 7f04457fa700 1 -- 172.21.15.3:0/817138760 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f043c0963e0 con 0x7f0448100910 2023-07-20T01:42:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.714+0000 7f044e73f700 1 -- 172.21.15.3:0/817138760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f044810b120 con 0x7f0448100910 2023-07-20T01:42:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.715+0000 7f04457fa700 1 -- 172.21.15.3:0/817138760 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8061 (secure 0 0 0) 0x7f043c099a00 con 0x7f0448100910 2023-07-20T01:42:25.682 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:25.682 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3626612235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3626612235},{"type":"v1","addr":"172.21.15.3:6835","nonce":3626612235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3145567212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3145567212},{"type":"v1","addr":"172.21.15.3:6837","nonce":3145567212}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/1529105586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":1529105586},{"type":"v1","addr":"172.21.15.3:6839","nonce":1529105586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6833/2663512295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2663512295},{"type":"v1","addr":"172.21.15.43:6833","nonce":2663512295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2824762692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2824762692},{"type":"v1","addr":"172.21.15.86:6833","nonce":2824762692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2898918243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2898918243},{"type":"v1","addr":"172.21.15.43:6835","nonce":2898918243}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/113908943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":113908943},{"type":"v1","addr":"172.21.15.86:6835","nonce":113908943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1379813430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1379813430},{"type":"v1","addr":"172.21.15.43:6837","nonce":1379813430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2620198024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2620198024},{"type":"v1","addr":"172.21.15.86:6837","nonce":2620198024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/1279793907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1279793907},{"type":"v1","addr":"172.21.15.86:6839","nonce":1279793907}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T01:42:25.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 -- 172.21.15.3:0/817138760 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f04300a1150 msgr2=0x7f04300a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:25.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f04300a1150 0x7f04300a3610 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0438005fd0 tx=0x7f0438005dc0 comp rx=0 tx=0).stop 2023-07-20T01:42:25.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 -- 172.21.15.3:0/817138760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0448100910 msgr2=0x7f0448101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:25.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0448100910 0x7f0448101cf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f043c00d950 tx=0x7f043c00dd10 comp rx=0 tx=0).stop 2023-07-20T01:42:25.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 -- 172.21.15.3:0/817138760 shutdown_connections 2023-07-20T01:42:25.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04481012b0 0x7f0448102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:25.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f04300a1150 0x7f04300a3610 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:25.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04480ff720 0x7f04481036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:25.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 --2- 172.21.15.3:0/817138760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0448100910 0x7f0448101cf0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:25.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 -- 172.21.15.3:0/817138760 >> 172.21.15.3:0/817138760 conn(0x7f04480facb0 msgr2=0x7f0448067a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:25.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 -- 172.21.15.3:0/817138760 shutdown_connections 2023-07-20T01:42:25.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:25.717+0000 7f044e73f700 1 -- 172.21.15.3:0/817138760 wait complete. 2023-07-20T01:42:25.686 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 11 2023-07-20T01:42:25.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:25 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:25.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:25 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:25.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:25 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:25.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:25 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:25.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:25 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:25.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:25 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:25.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:25 smithi003 ceph-mon[96422]: Deploying daemon mds.a on smithi003 2023-07-20T01:42:25.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:25 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.3:6838/1529105586,v1:172.21.15.3:6839/1529105586] up:boot 2023-07-20T01:42:25.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:25 smithi003 ceph-mon[96422]: fsmap 10 up:standby 2023-07-20T01:42:25.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:25 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T01:42:25.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:25 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:25.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:25 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:25.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:25 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:25.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:25 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:25.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:25 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:25.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:25 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:25.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:25 smithi043 ceph-mon[97823]: Deploying daemon mds.a on smithi003 2023-07-20T01:42:25.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:25 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.3:6838/1529105586,v1:172.21.15.3:6839/1529105586] up:boot 2023-07-20T01:42:25.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:25 smithi043 ceph-mon[97823]: fsmap 10 up:standby 2023-07-20T01:42:25.754 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:25 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T01:42:25.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:25 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:25.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:25 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:25.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:25 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:25.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:25 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:25.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:25 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:25.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:25 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:25.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:25 smithi086 ceph-mon[98138]: Deploying daemon mds.a on smithi003 2023-07-20T01:42:25.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:25 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.3:6838/1529105586,v1:172.21.15.3:6839/1529105586] up:boot 2023-07-20T01:42:25.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:25 smithi086 ceph-mon[98138]: fsmap 10 up:standby 2023-07-20T01:42:25.854 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:25 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T01:42:26.368 INFO:journalctl@ceph.mds.a.smithi003.stdout:Jul 20 01:42:26 smithi003 systemd[1]: Starting Ceph mds.a for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:26.635 INFO:journalctl@ceph.mds.a.smithi003.stdout:Jul 20 01:42:26 smithi003 podman[131301]: 2023-07-20T01:42:26.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:26 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/817138760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:26.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:26 smithi003 ceph-mon[96422]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:26.751 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:26 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/817138760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:26.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:26 smithi043 ceph-mon[97823]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:26.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:26 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/817138760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:26.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:26 smithi086 ceph-mon[98138]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:26.936 INFO:journalctl@ceph.mds.a.smithi003.stdout:Jul 20 01:42:26 smithi003 ceph-mds[131341]: -- 172.21.15.3:0/2316908160 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557e2ada31e0 con 0x557e2ad78800 2023-07-20T01:42:26.937 INFO:journalctl@ceph.mds.a.smithi003.stdout:Jul 20 01:42:26 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-a[131317]: starting mds.a at 2023-07-20T01:42:26.937 INFO:journalctl@ceph.mds.a.smithi003.stdout:Jul 20 01:42:26 smithi003 bash[131301]: dae3ff6a35b9441968874ee59bc0a951ef15e3b2583c285ffe71cbd084ab415d 2023-07-20T01:42:26.937 INFO:journalctl@ceph.mds.a.smithi003.stdout:Jul 20 01:42:26 smithi003 systemd[1]: Started Ceph mds.a for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:27.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:27.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.876+0000 7f4be1ca5700 1 -- 172.21.15.3:0/3807802215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4bdc101a20 msgr2=0x7f4bdc101e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:27.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.876+0000 7f4be1ca5700 1 --2- 172.21.15.3:0/3807802215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4bdc101a20 0x7f4bdc101e80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00099c0 tx=0x7f4bd0009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:27.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.876+0000 7f4be1ca5700 1 -- 172.21.15.3:0/3807802215 shutdown_connections 2023-07-20T01:42:27.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.876+0000 7f4be1ca5700 1 --2- 172.21.15.3:0/3807802215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bdc1023c0 0x7f4bdc108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:27.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.876+0000 7f4be1ca5700 1 --2- 172.21.15.3:0/3807802215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bdc100830 0x7f4bdc100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:27.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.876+0000 7f4be1ca5700 1 --2- 172.21.15.3:0/3807802215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4bdc101a20 0x7f4bdc101e80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:27.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.876+0000 7f4be1ca5700 1 -- 172.21.15.3:0/3807802215 >> 172.21.15.3:0/3807802215 conn(0x7f4bdc0fbdc0 msgr2=0x7f4bdc0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:27.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.877+0000 7f4be1ca5700 1 -- 172.21.15.3:0/3807802215 shutdown_connections 2023-07-20T01:42:27.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.877+0000 7f4be1ca5700 1 -- 172.21.15.3:0/3807802215 wait complete. 2023-07-20T01:42:27.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.877+0000 7f4be1ca5700 1 Processor -- start 2023-07-20T01:42:27.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4be1ca5700 1 -- start start 2023-07-20T01:42:27.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4be1ca5700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bdc100830 0x7f4bdc196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:27.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4be1ca5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bdc101a20 0x7f4bdc196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:27.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4be1ca5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4bdc1023c0 0x7f4bdc19dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:27.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4be1ca5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc19e0b0 con 0x7f4bdc1023c0 2023-07-20T01:42:27.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4be1ca5700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc19e1f0 con 0x7f4bdc100830 2023-07-20T01:42:27.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4be1ca5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc19e380 con 0x7f4bdc101a20 2023-07-20T01:42:27.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4bdc1023c0 0x7f4bdc19dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdb7fe700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bdc100830 0x7f4bdc196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdb7fe700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bdc100830 0x7f4bdc196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.3:45468/0 (socket says 172.21.15.3:45468) 2023-07-20T01:42:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdb7fe700 1 -- 172.21.15.3:0/2166436352 learned_addr learned my addr 172.21.15.3:0/2166436352 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdaffd700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bdc101a20 0x7f4bdc196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdbfff700 1 -- 172.21.15.3:0/2166436352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bdc101a20 msgr2=0x7f4bdc196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdbfff700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bdc101a20 0x7f4bdc196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdbfff700 1 -- 172.21.15.3:0/2166436352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bdc100830 msgr2=0x7f4bdc196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:27.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdbfff700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bdc100830 0x7f4bdc196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:27.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdbfff700 1 -- 172.21.15.3:0/2166436352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd00095f0 con 0x7f4bdc1023c0 2023-07-20T01:42:27.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdb7fe700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bdc100830 0x7f4bdc196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:42:27.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.878+0000 7f4bdbfff700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4bdc1023c0 0x7f4bdc19dae0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400efc0 tx=0x7f4bd400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:27.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.879+0000 7f4bd8ff9700 1 -- 172.21.15.3:0/2166436352 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4018070 con 0x7f4bdc1023c0 2023-07-20T01:42:27.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.879+0000 7f4bd8ff9700 1 -- 172.21.15.3:0/2166436352 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4bd400f040 con 0x7f4bdc1023c0 2023-07-20T01:42:27.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.879+0000 7f4be1ca5700 1 -- 172.21.15.3:0/2166436352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc19e600 con 0x7f4bdc1023c0 2023-07-20T01:42:27.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.879+0000 7f4bd8ff9700 1 -- 172.21.15.3:0/2166436352 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd400cd40 con 0x7f4bdc1023c0 2023-07-20T01:42:27.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.879+0000 7f4be1ca5700 1 -- 172.21.15.3:0/2166436352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc19eac0 con 0x7f4bdc1023c0 2023-07-20T01:42:27.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.880+0000 7f4be1ca5700 1 -- 172.21.15.3:0/2166436352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc10c090 con 0x7f4bdc1023c0 2023-07-20T01:42:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.881+0000 7f4bd8ff9700 1 -- 172.21.15.3:0/2166436352 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f4bd4003a00 con 0x7f4bdc1023c0 2023-07-20T01:42:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.882+0000 7f4bd8ff9700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4bc40a1150 0x7f4bc40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.882+0000 7f4bdb7fe700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4bc40a1150 0x7f4bc40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.882+0000 7f4bd8ff9700 1 -- 172.21.15.3:0/2166436352 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4bd40cc9e0 con 0x7f4bdc1023c0 2023-07-20T01:42:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.882+0000 7f4bdb7fe700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4bc40a1150 0x7f4bc40a3610 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0105c0 tx=0x7f4bcc00b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:27.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:27.885+0000 7f4bd8ff9700 1 -- 172.21.15.3:0/2166436352 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4bd4096660 con 0x7f4bdc1023c0 2023-07-20T01:42:28.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.128+0000 7f4be1ca5700 1 -- 172.21.15.3:0/2166436352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bdc19f100 con 0x7f4bdc1023c0 2023-07-20T01:42:28.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.128+0000 7f4bd8ff9700 1 -- 172.21.15.3:0/2166436352 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8816 (secure 0 0 0) 0x7f4bd4099c80 con 0x7f4bdc1023c0 2023-07-20T01:42:28.097 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:28.097 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3626612235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3626612235},{"type":"v1","addr":"172.21.15.3:6835","nonce":3626612235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3145567212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3145567212},{"type":"v1","addr":"172.21.15.3:6837","nonce":3145567212}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/1529105586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":1529105586},{"type":"v1","addr":"172.21.15.3:6839","nonce":1529105586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14619,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/2753440897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":2753440897},{"type":"v1","addr":"172.21.15.3:6841","nonce":2753440897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6833/2663512295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2663512295},{"type":"v1","addr":"172.21.15.43:6833","nonce":2663512295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2824762692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2824762692},{"type":"v1","addr":"172.21.15.86:6833","nonce":2824762692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2898918243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2898918243},{"type":"v1","addr":"172.21.15.43:6835","nonce":2898918243}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/113908943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":113908943},{"type":"v1","addr":"172.21.15.86:6835","nonce":113908943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1379813430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1379813430},{"type":"v1","addr":"172.21.15.43:6837","nonce":1379813430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2620198024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2620198024},{"type":"v1","addr":"172.21.15.86:6837","nonce":2620198024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/1279793907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1279793907},{"type":"v1","addr":"172.21.15.86:6839","nonce":1279793907}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T01:42:28.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.130+0000 7f4be1ca5700 1 -- 172.21.15.3:0/2166436352 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4bc40a1150 msgr2=0x7f4bc40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:28.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.130+0000 7f4be1ca5700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4bc40a1150 0x7f4bc40a3610 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0105c0 tx=0x7f4bcc00b4a0 comp rx=0 tx=0).stop 2023-07-20T01:42:28.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.130+0000 7f4be1ca5700 1 -- 172.21.15.3:0/2166436352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4bdc1023c0 msgr2=0x7f4bdc19dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:28.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.130+0000 7f4be1ca5700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4bdc1023c0 0x7f4bdc19dae0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400efc0 tx=0x7f4bd400c3a0 comp rx=0 tx=0).stop 2023-07-20T01:42:28.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.130+0000 7f4be1ca5700 1 -- 172.21.15.3:0/2166436352 shutdown_connections 2023-07-20T01:42:28.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.130+0000 7f4be1ca5700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4bdc1023c0 0x7f4bdc19dae0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:28.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.130+0000 7f4be1ca5700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f4bc40a1150 0x7f4bc40a3610 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:28.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.130+0000 7f4be1ca5700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4bdc100830 0x7f4bdc196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:28.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.130+0000 7f4be1ca5700 1 --2- 172.21.15.3:0/2166436352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bdc101a20 0x7f4bdc196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:28.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.130+0000 7f4be1ca5700 1 -- 172.21.15.3:0/2166436352 >> 172.21.15.3:0/2166436352 conn(0x7f4bdc0fbdc0 msgr2=0x7f4bdc0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:28.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.131+0000 7f4be1ca5700 1 -- 172.21.15.3:0/2166436352 shutdown_connections 2023-07-20T01:42:28.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:28.131+0000 7f4be1ca5700 1 -- 172.21.15.3:0/2166436352 wait complete. 2023-07-20T01:42:28.103 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 12 2023-07-20T01:42:28.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:28.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:28.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:28.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:28.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:28.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:28.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:27 smithi086 ceph-mon[98138]: Deploying daemon mds.b on smithi043 2023-07-20T01:42:28.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:27 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.3:6840/2753440897,v1:172.21.15.3:6841/2753440897] up:boot 2023-07-20T01:42:28.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:27 smithi086 ceph-mon[98138]: fsmap 11 up:standby 2023-07-20T01:42:28.106 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:27 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T01:42:28.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:28.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:28.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:28.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:28.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:28.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:28.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:27 smithi003 ceph-mon[96422]: Deploying daemon mds.b on smithi043 2023-07-20T01:42:28.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:27 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.3:6840/2753440897,v1:172.21.15.3:6841/2753440897] up:boot 2023-07-20T01:42:28.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:27 smithi003 ceph-mon[96422]: fsmap 11 up:standby 2023-07-20T01:42:28.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:27 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T01:42:28.185 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:27 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:28.185 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:27 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:28.185 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:27 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:28.186 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:27 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:42:28.186 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:27 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:42:28.186 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:27 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:28.187 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:27 smithi043 ceph-mon[97823]: Deploying daemon mds.b on smithi043 2023-07-20T01:42:28.187 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:27 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.3:6840/2753440897,v1:172.21.15.3:6841/2753440897] up:boot 2023-07-20T01:42:28.187 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:27 smithi043 ceph-mon[97823]: fsmap 11 up:standby 2023-07-20T01:42:28.187 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:27 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T01:42:28.722 INFO:journalctl@ceph.mds.b.smithi043.stdout:Jul 20 01:42:28 smithi043 systemd[1]: Starting Ceph mds.b for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:29.001 INFO:journalctl@ceph.mds.b.smithi043.stdout:Jul 20 01:42:28 smithi043 podman[123086]: 2023-07-20T01:42:29.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:28 smithi043 ceph-mon[97823]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:29.002 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:28 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2166436352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:29.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:28 smithi086 ceph-mon[98138]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:29.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:28 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2166436352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:29.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:28 smithi003 ceph-mon[96422]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:29.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:28 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2166436352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:29.501 INFO:journalctl@ceph.mds.b.smithi043.stdout:Jul 20 01:42:29 smithi043 ceph-mds[123126]: -- 172.21.15.43:0/2501188306 <== mon.2 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cec3a4d1e0 con 0x55cec3a22800 2023-07-20T01:42:29.502 INFO:journalctl@ceph.mds.b.smithi043.stdout:Jul 20 01:42:29 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-b[123102]: starting mds.b at 2023-07-20T01:42:29.502 INFO:journalctl@ceph.mds.b.smithi043.stdout:Jul 20 01:42:29 smithi043 bash[123086]: eed0082163df30f34181f3bda3d9e3a422ac5f982501dc2d88ea0721c8ca5ce6 2023-07-20T01:42:29.502 INFO:journalctl@ceph.mds.b.smithi043.stdout:Jul 20 01:42:29 smithi043 systemd[1]: Started Ceph mds.b for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:30.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:42:30.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.279+0000 7f322681f700 1 -- 172.21.15.3:0/4073854426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32201065b0 msgr2=0x7f32201089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:30.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.279+0000 7f322681f700 1 --2- 172.21.15.3:0/4073854426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32201065b0 0x7f32201089a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f321800b0a0 tx=0x7f321800b3b0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.279+0000 7f322681f700 1 -- 172.21.15.3:0/4073854426 shutdown_connections 2023-07-20T01:42:30.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.279+0000 7f322681f700 1 --2- 172.21.15.3:0/4073854426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32201065b0 0x7f32201089a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.279+0000 7f322681f700 1 --2- 172.21.15.3:0/4073854426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3220101350 0x7f3220103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.279+0000 7f322681f700 1 --2- 172.21.15.3:0/4073854426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3220103c80 0x7f3220106070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.279+0000 7f322681f700 1 -- 172.21.15.3:0/4073854426 >> 172.21.15.3:0/4073854426 conn(0x7f32200faca0 msgr2=0x7f32200fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:30.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.279+0000 7f322681f700 1 -- 172.21.15.3:0/4073854426 shutdown_connections 2023-07-20T01:42:30.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.279+0000 7f322681f700 1 -- 172.21.15.3:0/4073854426 wait complete. 2023-07-20T01:42:30.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.279+0000 7f322681f700 1 Processor -- start 2023-07-20T01:42:30.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.279+0000 7f322681f700 1 -- start start 2023-07-20T01:42:30.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f322681f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3220101350 0x7f3220100cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:30.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f322681f700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3220103c80 0x7f32200ff300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:30.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f322681f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32201065b0 0x7f32200ff840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:30.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f322681f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32200fffe0 con 0x7f3220101350 2023-07-20T01:42:30.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f322681f700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3220100150 con 0x7f3220103c80 2023-07-20T01:42:30.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f322681f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32201a0670 con 0x7f32201065b0 2023-07-20T01:42:30.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3220101350 0x7f3220100cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:30.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3220101350 0x7f3220100cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42662/0 (socket says 172.21.15.3:42662) 2023-07-20T01:42:30.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321ffff700 1 -- 172.21.15.3:0/4074249930 learned_addr learned my addr 172.21.15.3:0/4074249930 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:30.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f3224dbc700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32201065b0 0x7f32200ff840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:30.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321f7fe700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3220103c80 0x7f32200ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:30.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321ffff700 1 -- 172.21.15.3:0/4074249930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32201065b0 msgr2=0x7f32200ff840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:30.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321ffff700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32201065b0 0x7f32200ff840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321ffff700 1 -- 172.21.15.3:0/4074249930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3220103c80 msgr2=0x7f32200ff300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:30.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321ffff700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3220103c80 0x7f32200ff300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321ffff700 1 -- 172.21.15.3:0/4074249930 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3218009d00 con 0x7f3220101350 2023-07-20T01:42:30.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321ffff700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3220101350 0x7f3220100cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f321000e9a0 tx=0x7f321000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:30.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321d7fa700 1 -- 172.21.15.3:0/4074249930 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3210009790 con 0x7f3220101350 2023-07-20T01:42:30.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f321d7fa700 1 -- 172.21.15.3:0/4074249930 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3210010430 con 0x7f3220101350 2023-07-20T01:42:30.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.280+0000 7f322681f700 1 -- 172.21.15.3:0/4074249930 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32201a0950 con 0x7f3220101350 2023-07-20T01:42:30.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.281+0000 7f321d7fa700 1 -- 172.21.15.3:0/4074249930 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3210018680 con 0x7f3220101350 2023-07-20T01:42:30.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.281+0000 7f322681f700 1 -- 172.21.15.3:0/4074249930 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32201a0e80 con 0x7f3220101350 2023-07-20T01:42:30.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.281+0000 7f322681f700 1 -- 172.21.15.3:0/4074249930 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32200fc820 con 0x7f3220101350 2023-07-20T01:42:30.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.283+0000 7f321d7fa700 1 -- 172.21.15.3:0/4074249930 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f32100105a0 con 0x7f3220101350 2023-07-20T01:42:30.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.283+0000 7f321d7fa700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f320c0a11d0 0x7f320c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:30.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.284+0000 7f321f7fe700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f320c0a11d0 0x7f320c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:30.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.284+0000 7f321d7fa700 1 -- 172.21.15.3:0/4074249930 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3210014070 con 0x7f3220101350 2023-07-20T01:42:30.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.284+0000 7f321f7fe700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f320c0a11d0 0x7f320c0a3690 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3208009c60 tx=0x7f3208009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:30.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.288+0000 7f321d7fa700 1 -- 172.21.15.3:0/4074249930 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f32100963d0 con 0x7f3220101350 2023-07-20T01:42:30.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:42:30.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:30 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.43:6838/3251120203,v1:172.21.15.43:6839/3251120203] up:boot 2023-07-20T01:42:30.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:30 smithi086 ceph-mon[98138]: fsmap 12 up:standby 2023-07-20T01:42:30.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:30 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T01:42:30.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:30 smithi086 ceph-mon[98138]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:30.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:42:30.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:30 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.43:6838/3251120203,v1:172.21.15.43:6839/3251120203] up:boot 2023-07-20T01:42:30.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:30 smithi003 ceph-mon[96422]: fsmap 12 up:standby 2023-07-20T01:42:30.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:30 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T01:42:30.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:30 smithi003 ceph-mon[96422]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:30.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:30.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:42:30.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:30 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.43:6838/3251120203,v1:172.21.15.43:6839/3251120203] up:boot 2023-07-20T01:42:30.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:30 smithi043 ceph-mon[97823]: fsmap 12 up:standby 2023-07-20T01:42:30.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:30 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T01:42:30.503 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:30 smithi043 ceph-mon[97823]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:30.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.542+0000 7f322681f700 1 -- 172.21.15.3:0/4074249930 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f322002c970 con 0x7f3220101350 2023-07-20T01:42:30.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.543+0000 7f321d7fa700 1 -- 172.21.15.3:0/4074249930 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9574 (secure 0 0 0) 0x7f32100999f0 con 0x7f3220101350 2023-07-20T01:42:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:30.511 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3626612235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3626612235},{"type":"v1","addr":"172.21.15.3:6835","nonce":3626612235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3145567212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3145567212},{"type":"v1","addr":"172.21.15.3:6837","nonce":3145567212}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/1529105586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":1529105586},{"type":"v1","addr":"172.21.15.3:6839","nonce":1529105586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14619,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/2753440897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":2753440897},{"type":"v1","addr":"172.21.15.3:6841","nonce":2753440897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6833/2663512295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2663512295},{"type":"v1","addr":"172.21.15.43:6833","nonce":2663512295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2824762692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2824762692},{"type":"v1","addr":"172.21.15.86:6833","nonce":2824762692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/2898918243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2898918243},{"type":"v1","addr":"172.21.15.43:6835","nonce":2898918243}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/113908943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":113908943},{"type":"v1","addr":"172.21.15.86:6835","nonce":113908943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1379813430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1379813430},{"type":"v1","addr":"172.21.15.43:6837","nonce":1379813430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2620198024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2620198024},{"type":"v1","addr":"172.21.15.86:6837","nonce":2620198024}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24494,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/3251120203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":3251120203},{"type":"v1","addr":"172.21.15.43:6839","nonce":3251120203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/1279793907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1279793907},{"type":"v1","addr":"172.21.15.86:6839","nonce":1279793907}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T01:42:30.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 -- 172.21.15.3:0/4074249930 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f320c0a11d0 msgr2=0x7f320c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:30.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f320c0a11d0 0x7f320c0a3690 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3208009c60 tx=0x7f3208009260 comp rx=0 tx=0).stop 2023-07-20T01:42:30.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 -- 172.21.15.3:0/4074249930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3220101350 msgr2=0x7f3220100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:30.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3220101350 0x7f3220100cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f321000e9a0 tx=0x7f321000ecb0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 -- 172.21.15.3:0/4074249930 shutdown_connections 2023-07-20T01:42:30.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32201065b0 0x7f32200ff840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3220101350 0x7f3220100cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f320c0a11d0 0x7f320c0a3690 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 --2- 172.21.15.3:0/4074249930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3220103c80 0x7f32200ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 -- 172.21.15.3:0/4074249930 >> 172.21.15.3:0/4074249930 conn(0x7f32200faca0 msgr2=0x7f3220101fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:30.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 -- 172.21.15.3:0/4074249930 shutdown_connections 2023-07-20T01:42:30.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.546+0000 7f322681f700 1 -- 172.21.15.3:0/4074249930 wait complete. 2023-07-20T01:42:30.515 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 13 2023-07-20T01:42:30.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T01:42:30.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.688+0000 7fdb46170700 1 -- 172.21.15.3:0/3792356465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb40100910 msgr2=0x7fdb40100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:30.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.688+0000 7fdb46170700 1 --2- 172.21.15.3:0/3792356465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb40100910 0x7fdb40100d70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdb280099c0 tx=0x7fdb28009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.688+0000 7fdb46170700 1 -- 172.21.15.3:0/3792356465 shutdown_connections 2023-07-20T01:42:30.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.688+0000 7fdb46170700 1 --2- 172.21.15.3:0/3792356465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb401012b0 0x7fdb40107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.688+0000 7fdb46170700 1 --2- 172.21.15.3:0/3792356465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb400ff720 0x7fdb400ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.688+0000 7fdb46170700 1 --2- 172.21.15.3:0/3792356465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb40100910 0x7fdb40100d70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.688+0000 7fdb46170700 1 -- 172.21.15.3:0/3792356465 >> 172.21.15.3:0/3792356465 conn(0x7fdb400facb0 msgr2=0x7fdb400fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:30.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.688+0000 7fdb46170700 1 -- 172.21.15.3:0/3792356465 shutdown_connections 2023-07-20T01:42:30.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.688+0000 7fdb46170700 1 -- 172.21.15.3:0/3792356465 wait complete. 2023-07-20T01:42:30.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb46170700 1 Processor -- start 2023-07-20T01:42:30.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb46170700 1 -- start start 2023-07-20T01:42:30.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb46170700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb400ff720 0x7fdb401036b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:30.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb46170700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb40100910 0x7fdb40101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:30.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb46170700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb401012b0 0x7fdb40102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:30.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb46170700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4019a350 con 0x7fdb400ff720 2023-07-20T01:42:30.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb46170700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb40102770 con 0x7fdb401012b0 2023-07-20T01:42:30.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb46170700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb40102930 con 0x7fdb40100910 2023-07-20T01:42:30.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb3f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb400ff720 0x7fdb401036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:30.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb3f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb400ff720 0x7fdb401036b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42676/0 (socket says 172.21.15.3:42676) 2023-07-20T01:42:30.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb3f7fe700 1 -- 172.21.15.3:0/2846500308 learned_addr learned my addr 172.21.15.3:0/2846500308 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:30.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.689+0000 7fdb3effd700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb40100910 0x7fdb40101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:30.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb3ffff700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb401012b0 0x7fdb40102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:30.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb3f7fe700 1 -- 172.21.15.3:0/2846500308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb40100910 msgr2=0x7fdb40101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:30.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb3f7fe700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb40100910 0x7fdb40101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb3f7fe700 1 -- 172.21.15.3:0/2846500308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb401012b0 msgr2=0x7fdb40102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:30.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb3f7fe700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb401012b0 0x7fdb40102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb3f7fe700 1 -- 172.21.15.3:0/2846500308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb280095f0 con 0x7fdb400ff720 2023-07-20T01:42:30.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb3f7fe700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb400ff720 0x7fdb401036b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000d980 tx=0x7fdb3000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:30.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb3cff9700 1 -- 172.21.15.3:0/2846500308 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3000b950 con 0x7fdb400ff720 2023-07-20T01:42:30.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb3cff9700 1 -- 172.21.15.3:0/2846500308 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb30007620 con 0x7fdb400ff720 2023-07-20T01:42:30.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb3cff9700 1 -- 172.21.15.3:0/2846500308 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3000f3c0 con 0x7fdb400ff720 2023-07-20T01:42:30.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb46170700 1 -- 172.21.15.3:0/2846500308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb401a4d00 con 0x7fdb400ff720 2023-07-20T01:42:30.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.690+0000 7fdb46170700 1 -- 172.21.15.3:0/2846500308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb401a5120 con 0x7fdb400ff720 2023-07-20T01:42:30.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.691+0000 7fdb46170700 1 -- 172.21.15.3:0/2846500308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4010af40 con 0x7fdb400ff720 2023-07-20T01:42:30.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.692+0000 7fdb3cff9700 1 -- 172.21.15.3:0/2846500308 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fdb30010640 con 0x7fdb400ff720 2023-07-20T01:42:30.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.693+0000 7fdb3cff9700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdb2c0a11d0 0x7fdb2c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:30.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.693+0000 7fdb3effd700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdb2c0a11d0 0x7fdb2c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:30.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.693+0000 7fdb3cff9700 1 -- 172.21.15.3:0/2846500308 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdb300cc290 con 0x7fdb400ff720 2023-07-20T01:42:30.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.693+0000 7fdb3effd700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdb2c0a11d0 0x7fdb2c0a3690 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800b580 tx=0x7fdb28005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:30.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.697+0000 7fdb3cff9700 1 -- 172.21.15.3:0/2846500308 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdb30095e80 con 0x7fdb400ff720 2023-07-20T01:42:30.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.907+0000 7fdb46170700 1 -- 172.21.15.3:0/2846500308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb401a56d0 con 0x7fdb400ff720 2023-07-20T01:42:30.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.908+0000 7fdb3cff9700 1 -- 172.21.15.3:0/2846500308 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18240 (secure 0 0 0) 0x7fdb300b9070 con 0x7fdb400ff720 2023-07-20T01:42:30.876 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:30.876 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":76,"fsid":"9b4fd2fc-269d-11ee-9b34-001a4aab830c","created":"2023-07-20T01:35:37.746283+0000","modified":"2023-07-20T01:41:54.557580+0000","last_up_change":"2023-07-20T01:41:53.551492+0000","last_in_change":"2023-07-20T01:41:35.988339+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T01:38:42.316631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0073d11-a48f-415c-babd-5672ef2585b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6803","nonce":1627689053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6805","nonce":1627689053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6809","nonce":1627689053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6807","nonce":1627689053}]},"public_addr":"172.21.15.3:6803/1627689053","cluster_addr":"172.21.15.3:6805/1627689053","heartbeat_back_addr":"172.21.15.3:6809/1627689053","heartbeat_front_addr":"172.21.15.3:6807/1627689053","state":["exists","up"]},{"osd":1,"uuid":"91f60b3d-e08c-4a41-b480-c3d30348eacc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6801","nonce":3632963249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6803","nonce":3632963249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6807","nonce":3632963249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6805","nonce":3632963249}]},"public_addr":"172.21.15.43:6801/3632963249","cluster_addr":"172.21.15.43:6803/3632963249","heartbeat_back_addr":"172.21.15.43:6807/3632963249","heartbeat_front_addr":"172.21.15.43:6805/3632963249","state":["exists","up"]},{"osd":2,"uuid":"11808f4b-c50a-44f4-8581-3fc226653b39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6801","nonce":2141120676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6803","nonce":2141120676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6807","nonce":2141120676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6805","nonce":2141120676}]},"public_addr":"172.21.15.86:6801/2141120676","cluster_addr":"172.21.15.86:6803/2141120676","heartbeat_back_addr":"172.21.15.86:6807/2141120676","heartbeat_front_addr":"172.21.15.86:6805/2141120676","state":["exists","up"]},{"osd":3,"uuid":"45c2d3f6-9f5d-4510-9dad-6d95d044725f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6811","nonce":535233193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6813","nonce":535233193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6817","nonce":535233193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6815","nonce":535233193}]},"public_addr":"172.21.15.3:6811/535233193","cluster_addr":"172.21.15.3:6813/535233193","heartbeat_back_addr":"172.21.15.3:6817/535233193","heartbeat_front_addr":"172.21.15.3:6815/535233193","state":["exists","up"]},{"osd":4,"uuid":"7a35bbae-6f6c-4296-9459-f1b4d2de632f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6809","nonce":47431618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6811","nonce":47431618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6815","nonce":47431618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6813","nonce":47431618}]},"public_addr":"172.21.15.43:6809/47431618","cluster_addr":"172.21.15.43:6811/47431618","heartbeat_back_addr":"172.21.15.43:6815/47431618","heartbeat_front_addr":"172.21.15.43:6813/47431618","state":["exists","up"]},{"osd":5,"uuid":"34cccdff-563b-413c-b56c-08839b1534aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6809","nonce":1573392100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1573392100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1573392100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1573392100}]},"public_addr":"172.21.15.86:6809/1573392100","cluster_addr":"172.21.15.86:6811/1573392100","heartbeat_back_addr":"172.21.15.86:6815/1573392100","heartbeat_front_addr":"172.21.15.86:6813/1573392100","state":["exists","up"]},{"osd":6,"uuid":"e1c8f2b2-aed6-4277-b645-6a68e287ca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6819","nonce":1511361132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6821","nonce":1511361132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6825","nonce":1511361132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6823","nonce":1511361132}]},"public_addr":"172.21.15.3:6819/1511361132","cluster_addr":"172.21.15.3:6821/1511361132","heartbeat_back_addr":"172.21.15.3:6825/1511361132","heartbeat_front_addr":"172.21.15.3:6823/1511361132","state":["exists","up"]},{"osd":7,"uuid":"fe6f3c63-9935-43b0-858f-11fd5f91c9ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6817","nonce":3536213360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6819","nonce":3536213360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6823","nonce":3536213360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6821","nonce":3536213360}]},"public_addr":"172.21.15.43:6817/3536213360","cluster_addr":"172.21.15.43:6819/3536213360","heartbeat_back_addr":"172.21.15.43:6823/3536213360","heartbeat_front_addr":"172.21.15.43:6821/3536213360","state":["exists","up"]},{"osd":8,"uuid":"7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6817","nonce":2823536777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6819","nonce":2823536777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6823","nonce":2823536777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6821","nonce":2823536777}]},"public_addr":"172.21.15.86:6817/2823536777","cluster_addr":"172.21.15.86:6819/2823536777","heartbeat_back_addr":"172.21.15.86:6823/2823536777","heartbeat_front_addr":"172.21.15.86:6821/2823536777","state":["exists","up"]},{"osd":9,"uuid":"d356aa99-85dc-4006-9441-e4f7d23d594f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6827","nonce":272617110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6829","nonce":272617110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6833","nonce":272617110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6831","nonce":272617110}]},"public_addr":"172.21.15.3:6827/272617110","cluster_addr":"172.21.15.3:6829/272617110","heartbeat_back_addr":"172.21.15.3:6833/272617110","heartbeat_front_addr":"172.21.15.3:6831/272617110","state":["exists","up"]},{"osd":10,"uuid":"b8848e6f-3991-4755-8fd3-627c9ebcd823","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6825","nonce":161869649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6827","nonce":161869649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6831","nonce":161869649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6829","nonce":161869649}]},"public_addr":"172.21.15.43:6825/161869649","cluster_addr":"172.21.15.43:6827/161869649","heartbeat_back_addr":"172.21.15.43:6831/161869649","heartbeat_front_addr":"172.21.15.43:6829/161869649","state":["exists","up"]},{"osd":11,"uuid":"e0fb9b4d-8559-48c2-9b41-c61a83ca0a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6825","nonce":3393537360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6827","nonce":3393537360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6831","nonce":3393537360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6829","nonce":3393537360}]},"public_addr":"172.21.15.86:6825/3393537360","cluster_addr":"172.21.15.86:6827/3393537360","heartbeat_back_addr":"172.21.15.86:6831/3393537360","heartbeat_front_addr":"172.21.15.86:6829/3393537360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:37:59.181659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:38:20.326807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:38:40.402049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:02.811605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:23.889505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:44.425791+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:06.236232+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:27.676116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:47.846169+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:09.462761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:31.165679+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:51.723548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/942763937":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6801/596448098":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6800/596448098":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6800/417953684":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/1023086410":"2023-07-21T01:36:35.772246+0000","172.21.15.3:0/3900831138":"2023-07-21T01:36:35.772246+0000","172.21.15.3:0/2570287892":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/286795203":"2023-07-21T01:36:10.337325+0000","172.21.15.3:6801/417953684":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/2757371565":"2023-07-21T01:36:10.337325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T01:42:30.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 -- 172.21.15.3:0/2846500308 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdb2c0a11d0 msgr2=0x7fdb2c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:30.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdb2c0a11d0 0x7fdb2c0a3690 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800b580 tx=0x7fdb28005bc0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 -- 172.21.15.3:0/2846500308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb400ff720 msgr2=0x7fdb401036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:30.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb400ff720 0x7fdb401036b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000d980 tx=0x7fdb3000dc90 comp rx=0 tx=0).stop 2023-07-20T01:42:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 -- 172.21.15.3:0/2846500308 shutdown_connections 2023-07-20T01:42:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fdb2c0a11d0 0x7fdb2c0a3690 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb401012b0 0x7fdb40102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb400ff720 0x7fdb401036b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 --2- 172.21.15.3:0/2846500308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb40100910 0x7fdb40101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 -- 172.21.15.3:0/2846500308 >> 172.21.15.3:0/2846500308 conn(0x7fdb400facb0 msgr2=0x7fdb40067a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 -- 172.21.15.3:0/2846500308 shutdown_connections 2023-07-20T01:42:30.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:30.911+0000 7fdb46170700 1 -- 172.21.15.3:0/2846500308 wait complete. 2023-07-20T01:42:30.892 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T01:38:42.316631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T01:42:30.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T01:42:31.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.057+0000 7efd07917700 1 -- 172.21.15.3:0/2898954716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00100930 msgr2=0x7efd00100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.057+0000 7efd07917700 1 --2- 172.21.15.3:0/2898954716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00100930 0x7efd00100d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efcf00099c0 tx=0x7efcf0009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.057+0000 7efd07917700 1 -- 172.21.15.3:0/2898954716 shutdown_connections 2023-07-20T01:42:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.057+0000 7efd07917700 1 --2- 172.21.15.3:0/2898954716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd001012d0 0x7efd00107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.057+0000 7efd07917700 1 --2- 172.21.15.3:0/2898954716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd000ff740 0x7efd000ffb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.057+0000 7efd07917700 1 --2- 172.21.15.3:0/2898954716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00100930 0x7efd00100d90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.057+0000 7efd07917700 1 -- 172.21.15.3:0/2898954716 >> 172.21.15.3:0/2898954716 conn(0x7efd000fac90 msgr2=0x7efd000fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:31.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.057+0000 7efd07917700 1 -- 172.21.15.3:0/2898954716 shutdown_connections 2023-07-20T01:42:31.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.057+0000 7efd07917700 1 -- 172.21.15.3:0/2898954716 wait complete. 2023-07-20T01:42:31.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd07917700 1 Processor -- start 2023-07-20T01:42:31.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd07917700 1 -- start start 2023-07-20T01:42:31.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd07917700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd000ff740 0x7efd00198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd07917700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd00100930 0x7efd00198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd07917700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd001012d0 0x7efd0019fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd07917700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd001a0310 con 0x7efd001012d0 2023-07-20T01:42:31.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd07917700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd001a0450 con 0x7efd000ff740 2023-07-20T01:42:31.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd07917700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd001a05e0 con 0x7efd00100930 2023-07-20T01:42:31.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd05eb4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd001012d0 0x7efd0019fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd04eb2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd00100930 0x7efd00198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd05eb4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd001012d0 0x7efd0019fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42694/0 (socket says 172.21.15.3:42694) 2023-07-20T01:42:31.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd05eb4700 1 -- 172.21.15.3:0/1885299988 learned_addr learned my addr 172.21.15.3:0/1885299988 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:31.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd056b3700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd000ff740 0x7efd00198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd05eb4700 1 -- 172.21.15.3:0/1885299988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd00100930 msgr2=0x7efd00198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd05eb4700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd00100930 0x7efd00198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd05eb4700 1 -- 172.21.15.3:0/1885299988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd000ff740 msgr2=0x7efd00198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd05eb4700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd000ff740 0x7efd00198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd05eb4700 1 -- 172.21.15.3:0/1885299988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf00095f0 con 0x7efd001012d0 2023-07-20T01:42:31.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.058+0000 7efd04eb2700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd00100930 0x7efd00198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:42:31.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.059+0000 7efd05eb4700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd001012d0 0x7efd0019fd40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efcf800ef30 tx=0x7efcf800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:31.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.059+0000 7efcf67fc700 1 -- 172.21.15.3:0/1885299988 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8018070 con 0x7efd001012d0 2023-07-20T01:42:31.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.059+0000 7efd07917700 1 -- 172.21.15.3:0/1885299988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd001a0860 con 0x7efd001012d0 2023-07-20T01:42:31.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.059+0000 7efcf67fc700 1 -- 172.21.15.3:0/1885299988 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcf800f040 con 0x7efd001012d0 2023-07-20T01:42:31.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.059+0000 7efcf67fc700 1 -- 172.21.15.3:0/1885299988 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf800cd00 con 0x7efd001012d0 2023-07-20T01:42:31.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.060+0000 7efd07917700 1 -- 172.21.15.3:0/1885299988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd001a0dc0 con 0x7efd001012d0 2023-07-20T01:42:31.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.060+0000 7efd07917700 1 -- 172.21.15.3:0/1885299988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd00109f00 con 0x7efd001012d0 2023-07-20T01:42:31.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.062+0000 7efcf67fc700 1 -- 172.21.15.3:0/1885299988 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7efcf8003a00 con 0x7efd001012d0 2023-07-20T01:42:31.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.062+0000 7efcf67fc700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efcec0a1150 0x7efcec0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.062+0000 7efcf67fc700 1 -- 172.21.15.3:0/1885299988 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efcf80cbc10 con 0x7efd001012d0 2023-07-20T01:42:31.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.062+0000 7efd056b3700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efcec0a1150 0x7efcec0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.063+0000 7efd056b3700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efcec0a1150 0x7efcec0a3610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efd00100790 tx=0x7efcfc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:31.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.067+0000 7efcf67fc700 1 -- 172.21.15.3:0/1885299988 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efcf8095890 con 0x7efd001012d0 2023-07-20T01:42:31.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:31 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/4074249930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:31.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:31 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:31.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:31 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2846500308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:31.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.292+0000 7efd07917700 1 -- 172.21.15.3:0/1885299988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd0010a0c0 con 0x7efd001012d0 2023-07-20T01:42:31.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.292+0000 7efcf67fc700 1 -- 172.21.15.3:0/1885299988 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd0010a0c0 con 0x7efd001012d0 2023-07-20T01:42:31.260 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-07-20T01:42:31.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 -- 172.21.15.3:0/1885299988 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efcec0a1150 msgr2=0x7efcec0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efcec0a1150 0x7efcec0a3610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efd00100790 tx=0x7efcfc009260 comp rx=0 tx=0).stop 2023-07-20T01:42:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 -- 172.21.15.3:0/1885299988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd001012d0 msgr2=0x7efd0019fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd001012d0 0x7efd0019fd40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efcf800ef30 tx=0x7efcf800c3a0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 -- 172.21.15.3:0/1885299988 shutdown_connections 2023-07-20T01:42:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd001012d0 0x7efd0019fd40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7efcec0a1150 0x7efcec0a3610 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd000ff740 0x7efd00198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 --2- 172.21.15.3:0/1885299988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd00100930 0x7efd00198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 -- 172.21.15.3:0/1885299988 >> 172.21.15.3:0/1885299988 conn(0x7efd000fac90 msgr2=0x7efd000fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:31.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 -- 172.21.15.3:0/1885299988 shutdown_connections 2023-07-20T01:42:31.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.294+0000 7efd07917700 1 -- 172.21.15.3:0/1885299988 wait complete. 2023-07-20T01:42:31.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-20T01:42:31.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:31 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/4074249930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:31.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:31 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:31.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:31 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2846500308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.436+0000 7f79735d8700 1 -- 172.21.15.3:0/4244171416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f796c101a20 msgr2=0x7f796c101e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.436+0000 7f79735d8700 1 --2- 172.21.15.3:0/4244171416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f796c101a20 0x7f796c101e80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f795c0099c0 tx=0x7f795c009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.436+0000 7f79735d8700 1 -- 172.21.15.3:0/4244171416 shutdown_connections 2023-07-20T01:42:31.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.437+0000 7f79735d8700 1 --2- 172.21.15.3:0/4244171416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796c1023c0 0x7f796c108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.437+0000 7f79735d8700 1 --2- 172.21.15.3:0/4244171416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f796c100830 0x7f796c100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.437+0000 7f79735d8700 1 --2- 172.21.15.3:0/4244171416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f796c101a20 0x7f796c101e80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.437+0000 7f79735d8700 1 -- 172.21.15.3:0/4244171416 >> 172.21.15.3:0/4244171416 conn(0x7f796c0fbdc0 msgr2=0x7f796c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:31.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.437+0000 7f79735d8700 1 -- 172.21.15.3:0/4244171416 shutdown_connections 2023-07-20T01:42:31.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.437+0000 7f79735d8700 1 -- 172.21.15.3:0/4244171416 wait complete. 2023-07-20T01:42:31.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.437+0000 7f79735d8700 1 Processor -- start 2023-07-20T01:42:31.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.437+0000 7f79735d8700 1 -- start start 2023-07-20T01:42:31.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.437+0000 7f79735d8700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f796c100830 0x7f796c1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.437+0000 7f79735d8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796c101a20 0x7f796c196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.437+0000 7f79735d8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f796c1023c0 0x7f796c19da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f79735d8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c19e050 con 0x7f796c1023c0 2023-07-20T01:42:31.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f79735d8700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c19e190 con 0x7f796c100830 2023-07-20T01:42:31.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f79735d8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c19e320 con 0x7f796c101a20 2023-07-20T01:42:31.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7971b75700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f796c1023c0 0x7f796c19da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7971374700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f796c100830 0x7f796c1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7971b75700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f796c1023c0 0x7f796c19da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42708/0 (socket says 172.21.15.3:42708) 2023-07-20T01:42:31.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7971374700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f796c100830 0x7f796c1964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.3:45524/0 (socket says 172.21.15.3:45524) 2023-07-20T01:42:31.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7971b75700 1 -- 172.21.15.3:0/2561492346 learned_addr learned my addr 172.21.15.3:0/2561492346 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:31.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7970b73700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796c101a20 0x7f796c196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7971b75700 1 -- 172.21.15.3:0/2561492346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796c101a20 msgr2=0x7f796c196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7971b75700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796c101a20 0x7f796c196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7971b75700 1 -- 172.21.15.3:0/2561492346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f796c100830 msgr2=0x7f796c1964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7971b75700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f796c100830 0x7f796c1964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7971b75700 1 -- 172.21.15.3:0/2561492346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c0095f0 con 0x7f796c1023c0 2023-07-20T01:42:31.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f7971b75700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f796c1023c0 0x7f796c19da80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f796400ef30 tx=0x7f796400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:31.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f79627fc700 1 -- 172.21.15.3:0/2561492346 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964018070 con 0x7f796c1023c0 2023-07-20T01:42:31.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f79627fc700 1 -- 172.21.15.3:0/2561492346 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f796400f040 con 0x7f796c1023c0 2023-07-20T01:42:31.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f79735d8700 1 -- 172.21.15.3:0/2561492346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c19e5a0 con 0x7f796c1023c0 2023-07-20T01:42:31.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f79627fc700 1 -- 172.21.15.3:0/2561492346 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796400cd00 con 0x7f796c1023c0 2023-07-20T01:42:31.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.438+0000 7f79735d8700 1 -- 172.21.15.3:0/2561492346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c19eab0 con 0x7f796c1023c0 2023-07-20T01:42:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.439+0000 7f79735d8700 1 -- 172.21.15.3:0/2561492346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796c107760 con 0x7f796c1023c0 2023-07-20T01:42:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.440+0000 7f79627fc700 1 -- 172.21.15.3:0/2561492346 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f7964003a00 con 0x7f796c1023c0 2023-07-20T01:42:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.441+0000 7f79627fc700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f79580a11d0 0x7f79580a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.441+0000 7f7971374700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f79580a11d0 0x7f79580a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.441+0000 7f79627fc700 1 -- 172.21.15.3:0/2561492346 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f79640cc850 con 0x7f796c1023c0 2023-07-20T01:42:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.441+0000 7f7971374700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f79580a11d0 0x7f79580a3690 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f796c101880 tx=0x7f7968007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.444+0000 7f79627fc700 1 -- 172.21.15.3:0/2561492346 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f79640964d0 con 0x7f796c1023c0 2023-07-20T01:42:31.501 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:31 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/4074249930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:42:31.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:31 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:31.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:31 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2846500308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:31.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.654+0000 7f79735d8700 1 -- 172.21.15.3:0/2561492346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f796c02c970 con 0x7f796c1023c0 2023-07-20T01:42:31.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.654+0000 7f79627fc700 1 -- 172.21.15.3:0/2561492346 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v76) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7964099af0 con 0x7f796c1023c0 2023-07-20T01:42:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:31.622 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2023-07-20T01:42:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-07-20T01:42:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2023-07-20T01:42:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": ".mgr" 2023-07-20T01:42:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-07-20T01:42:31.623 INFO:teuthology.orchestra.run.smithi003.stdout:] 2023-07-20T01:42:31.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.656+0000 7f79735d8700 1 -- 172.21.15.3:0/2561492346 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f79580a11d0 msgr2=0x7f79580a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.656+0000 7f79735d8700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f79580a11d0 0x7f79580a3690 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f796c101880 tx=0x7f7968007400 comp rx=0 tx=0).stop 2023-07-20T01:42:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.657+0000 7f79735d8700 1 -- 172.21.15.3:0/2561492346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f796c1023c0 msgr2=0x7f796c19da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.657+0000 7f79735d8700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f796c1023c0 0x7f796c19da80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f796400ef30 tx=0x7f796400c3a0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.657+0000 7f79735d8700 1 -- 172.21.15.3:0/2561492346 shutdown_connections 2023-07-20T01:42:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.657+0000 7f79735d8700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f796c1023c0 0x7f796c19da80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.657+0000 7f79735d8700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f79580a11d0 0x7f79580a3690 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.657+0000 7f79735d8700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f796c100830 0x7f796c1964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.657+0000 7f79735d8700 1 --2- 172.21.15.3:0/2561492346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796c101a20 0x7f796c196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.657+0000 7f79735d8700 1 -- 172.21.15.3:0/2561492346 >> 172.21.15.3:0/2561492346 conn(0x7f796c0fbdc0 msgr2=0x7f796c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:31.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.657+0000 7f79735d8700 1 -- 172.21.15.3:0/2561492346 shutdown_connections 2023-07-20T01:42:31.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.657+0000 7f79735d8700 1 -- 172.21.15.3:0/2561492346 wait complete. 2023-07-20T01:42:31.637 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-20T01:42:31.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-20T01:42:31.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.804+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3813184550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b9c1012e0 msgr2=0x7f2b9c107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.804+0000 7f2ba27a2700 1 --2- 172.21.15.3:0/3813184550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b9c1012e0 0x7f2b9c107b90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400b0d0 tx=0x7f2b9400b3e0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.805+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3813184550 shutdown_connections 2023-07-20T01:42:31.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.805+0000 7f2ba27a2700 1 --2- 172.21.15.3:0/3813184550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b9c1012e0 0x7f2b9c107b90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.805+0000 7f2ba27a2700 1 --2- 172.21.15.3:0/3813184550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b9c0ff750 0x7f2b9c0ffb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.805+0000 7f2ba27a2700 1 --2- 172.21.15.3:0/3813184550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9c100940 0x7f2b9c100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.805+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3813184550 >> 172.21.15.3:0/3813184550 conn(0x7f2b9c0faca0 msgr2=0x7f2b9c0fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:31.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.805+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3813184550 shutdown_connections 2023-07-20T01:42:31.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.805+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3813184550 wait complete. 2023-07-20T01:42:31.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.805+0000 7f2ba27a2700 1 Processor -- start 2023-07-20T01:42:31.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.805+0000 7f2ba27a2700 1 -- start start 2023-07-20T01:42:31.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2ba27a2700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b9c0ff750 0x7f2b9c1036e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2ba27a2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b9c100940 0x7f2b9c101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2ba27a2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9c1012e0 0x7f2b9c102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2ba27a2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c102a00 con 0x7f2b9c100940 2023-07-20T01:42:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2ba27a2700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c102b70 con 0x7f2b9c0ff750 2023-07-20T01:42:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2ba27a2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c1a06a0 con 0x7f2b9c1012e0 2023-07-20T01:42:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b9b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b9c100940 0x7f2b9c101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b9b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b9c100940 0x7f2b9c101d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42720/0 (socket says 172.21.15.3:42720) 2023-07-20T01:42:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b9b7fe700 1 -- 172.21.15.3:0/3059324376 learned_addr learned my addr 172.21.15.3:0/3059324376 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2ba0d3f700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9c1012e0 0x7f2b9c102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b9bfff700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b9c0ff750 0x7f2b9c1036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b9b7fe700 1 -- 172.21.15.3:0/3059324376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9c1012e0 msgr2=0x7f2b9c102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b9b7fe700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9c1012e0 0x7f2b9c102260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b9b7fe700 1 -- 172.21.15.3:0/3059324376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b9c0ff750 msgr2=0x7f2b9c1036e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:31.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b9b7fe700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b9c0ff750 0x7f2b9c1036e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:31.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b9b7fe700 1 -- 172.21.15.3:0/3059324376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b94009d00 con 0x7f2b9c100940 2023-07-20T01:42:31.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2ba0d3f700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9c1012e0 0x7f2b9c102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:42:31.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b9b7fe700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b9c100940 0x7f2b9c101d20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000e9c0 tx=0x7f2b9000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:31.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b997fa700 1 -- 172.21.15.3:0/3059324376 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b900097b0 con 0x7f2b9c100940 2023-07-20T01:42:31.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3059324376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9c1a0980 con 0x7f2b9c100940 2023-07-20T01:42:31.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b997fa700 1 -- 172.21.15.3:0/3059324376 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b9000ee60 con 0x7f2b9c100940 2023-07-20T01:42:31.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2b997fa700 1 -- 172.21.15.3:0/3059324376 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b900054f0 con 0x7f2b9c100940 2023-07-20T01:42:31.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.806+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3059324376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c1a0e40 con 0x7f2b9c100940 2023-07-20T01:42:31.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.807+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3059324376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9c10afe0 con 0x7f2b9c100940 2023-07-20T01:42:31.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.809+0000 7f2b997fa700 1 -- 172.21.15.3:0/3059324376 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f2b90004750 con 0x7f2b9c100940 2023-07-20T01:42:31.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.810+0000 7f2b997fa700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2b840a1150 0x7f2b840a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:31.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.810+0000 7f2b9bfff700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2b840a1150 0x7f2b840a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:31.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.810+0000 7f2b997fa700 1 -- 172.21.15.3:0/3059324376 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2b90014070 con 0x7f2b9c100940 2023-07-20T01:42:31.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.810+0000 7f2b9bfff700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2b840a1150 0x7f2b840a3610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c1007a0 tx=0x7f2b8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:31.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:31.813+0000 7f2b997fa700 1 -- 172.21.15.3:0/3059324376 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2b900964a0 con 0x7f2b9c100940 2023-07-20T01:42:32.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:32.049+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3059324376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2b9c10b1d0 con 0x7f2b9c100940 2023-07-20T01:42:32.751 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:32 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/1885299988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:42:32.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:32 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2561492346' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T01:42:32.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:32 smithi043 ceph-mon[97823]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:32.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:32 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/1885299988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:42:32.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:32 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2561492346' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T01:42:32.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:32 smithi086 ceph-mon[98138]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:32.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:32 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/1885299988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:42:32.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:32 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2561492346' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T01:42:32.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:32 smithi003 ceph-mon[96422]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:33.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.054+0000 7f2b997fa700 1 -- 172.21.15.3:0/3059324376 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2b90099ac0 con 0x7f2b9c100940 2023-07-20T01:42:33.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.058+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3059324376 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2b840a1150 msgr2=0x7f2b840a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:33.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.058+0000 7f2ba27a2700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2b840a1150 0x7f2b840a3610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c1007a0 tx=0x7f2b8c009260 comp rx=0 tx=0).stop 2023-07-20T01:42:33.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.058+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3059324376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b9c100940 msgr2=0x7f2b9c101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:33.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.058+0000 7f2ba27a2700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b9c100940 0x7f2b9c101d20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000e9c0 tx=0x7f2b9000ecd0 comp rx=0 tx=0).stop 2023-07-20T01:42:33.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.058+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3059324376 shutdown_connections 2023-07-20T01:42:33.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.058+0000 7f2ba27a2700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9c1012e0 0x7f2b9c102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:33.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.058+0000 7f2ba27a2700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f2b840a1150 0x7f2b840a3610 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c1007a0 tx=0x7f2b8c009260 comp rx=0 tx=0).stop 2023-07-20T01:42:33.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.058+0000 7f2ba27a2700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b9c0ff750 0x7f2b9c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:33.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.058+0000 7f2ba27a2700 1 --2- 172.21.15.3:0/3059324376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b9c100940 0x7f2b9c101d20 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000e9c0 tx=0x7f2b9000ecd0 comp rx=0 tx=0).stop 2023-07-20T01:42:33.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.058+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3059324376 >> 172.21.15.3:0/3059324376 conn(0x7f2b9c0faca0 msgr2=0x7f2b9c0687e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:33.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.059+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3059324376 shutdown_connections 2023-07-20T01:42:33.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.059+0000 7f2ba27a2700 1 -- 172.21.15.3:0/3059324376 wait complete. 2023-07-20T01:42:33.028 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2023-07-20T01:42:33.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-07-20T01:42:33.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.207+0000 7f7a89169700 1 -- 172.21.15.3:0/171803656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a84100860 msgr2=0x7f7a84100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.207+0000 7f7a89169700 1 --2- 172.21.15.3:0/171803656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a84100860 0x7f7a84100c80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400b0d0 tx=0x7f7a7400b3e0 comp rx=0 tx=0).stop 2023-07-20T01:42:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.207+0000 7f7a89169700 1 -- 172.21.15.3:0/171803656 shutdown_connections 2023-07-20T01:42:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.207+0000 7f7a89169700 1 --2- 172.21.15.3:0/171803656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a841023f0 0x7f7a84108ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.207+0000 7f7a89169700 1 --2- 172.21.15.3:0/171803656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a84100860 0x7f7a84100c80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.207+0000 7f7a89169700 1 --2- 172.21.15.3:0/171803656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a84101a50 0x7f7a84101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.207+0000 7f7a89169700 1 -- 172.21.15.3:0/171803656 >> 172.21.15.3:0/171803656 conn(0x7f7a840fbdd0 msgr2=0x7f7a840fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.207+0000 7f7a89169700 1 -- 172.21.15.3:0/171803656 shutdown_connections 2023-07-20T01:42:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.207+0000 7f7a89169700 1 -- 172.21.15.3:0/171803656 wait complete. 2023-07-20T01:42:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.208+0000 7f7a89169700 1 Processor -- start 2023-07-20T01:42:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.208+0000 7f7a89169700 1 -- start start 2023-07-20T01:42:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.208+0000 7f7a89169700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a84100860 0x7f7a84196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.208+0000 7f7a89169700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a84101a50 0x7f7a84196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.208+0000 7f7a89169700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a841023f0 0x7f7a8419dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.208+0000 7f7a89169700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8419e1f0 con 0x7f7a84101a50 2023-07-20T01:42:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.208+0000 7f7a89169700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8419e330 con 0x7f7a841023f0 2023-07-20T01:42:33.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.208+0000 7f7a89169700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8419e4c0 con 0x7f7a84100860 2023-07-20T01:42:33.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.208+0000 7f7a8259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a84101a50 0x7f7a84196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a8259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a84101a50 0x7f7a84196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42734/0 (socket says 172.21.15.3:42734) 2023-07-20T01:42:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a8259c700 1 -- 172.21.15.3:0/2866791918 learned_addr learned my addr 172.21.15.3:0/2866791918 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a82d9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a84100860 0x7f7a84196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a8359e700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a841023f0 0x7f7a8419dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a8259c700 1 -- 172.21.15.3:0/2866791918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a84100860 msgr2=0x7f7a84196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:33.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a8259c700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a84100860 0x7f7a84196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a8259c700 1 -- 172.21.15.3:0/2866791918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a841023f0 msgr2=0x7f7a8419dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a8259c700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a841023f0 0x7f7a8419dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a8259c700 1 -- 172.21.15.3:0/2866791918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a74009d00 con 0x7f7a84101a50 2023-07-20T01:42:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a82d9d700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a84100860 0x7f7a84196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:42:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a8259c700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a84101a50 0x7f7a84196bb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00e9a0 tx=0x7f7a6c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:33.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a7bfff700 1 -- 172.21.15.3:0/2866791918 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c009790 con 0x7f7a84101a50 2023-07-20T01:42:33.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a7bfff700 1 -- 172.21.15.3:0/2866791918 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a6c010430 con 0x7f7a84101a50 2023-07-20T01:42:33.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a89169700 1 -- 172.21.15.3:0/2866791918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8419e7a0 con 0x7f7a84101a50 2023-07-20T01:42:33.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a7bfff700 1 -- 172.21.15.3:0/2866791918 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c018680 con 0x7f7a84101a50 2023-07-20T01:42:33.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.209+0000 7f7a89169700 1 -- 172.21.15.3:0/2866791918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8419ecb0 con 0x7f7a84101a50 2023-07-20T01:42:33.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.210+0000 7f7a89169700 1 -- 172.21.15.3:0/2866791918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a84107da0 con 0x7f7a84101a50 2023-07-20T01:42:33.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.211+0000 7f7a7bfff700 1 -- 172.21.15.3:0/2866791918 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f7a6c0105a0 con 0x7f7a84101a50 2023-07-20T01:42:33.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.212+0000 7f7a7bfff700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f7a700a1150 0x7f7a700a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:33.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.212+0000 7f7a82d9d700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f7a700a1150 0x7f7a700a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:33.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.212+0000 7f7a7bfff700 1 -- 172.21.15.3:0/2866791918 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f7a6c014070 con 0x7f7a84101a50 2023-07-20T01:42:33.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.212+0000 7f7a82d9d700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f7a700a1150 0x7f7a700a3610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009fd0 tx=0x7f7a7400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:33.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.216+0000 7f7a7bfff700 1 -- 172.21.15.3:0/2866791918 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7a6c096410 con 0x7f7a84101a50 2023-07-20T01:42:33.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:33.422+0000 7f7a89169700 1 -- 172.21.15.3:0/2866791918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7a8402c970 con 0x7f7a84101a50 2023-07-20T01:42:34.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.061+0000 7f7a7bfff700 1 -- 172.21.15.3:0/2866791918 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v78) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7a6c099a30 con 0x7f7a84101a50 2023-07-20T01:42:34.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.074+0000 7f7a89169700 1 -- 172.21.15.3:0/2866791918 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f7a700a1150 msgr2=0x7f7a700a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:34.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.074+0000 7f7a89169700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f7a700a1150 0x7f7a700a3610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009fd0 tx=0x7f7a7400b040 comp rx=0 tx=0).stop 2023-07-20T01:42:34.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.074+0000 7f7a89169700 1 -- 172.21.15.3:0/2866791918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a84101a50 msgr2=0x7f7a84196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:34.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.074+0000 7f7a89169700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a84101a50 0x7f7a84196bb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00e9a0 tx=0x7f7a6c00ecb0 comp rx=0 tx=0).stop 2023-07-20T01:42:34.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.075+0000 7f7a89169700 1 -- 172.21.15.3:0/2866791918 shutdown_connections 2023-07-20T01:42:34.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.075+0000 7f7a89169700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f7a700a1150 0x7f7a700a3610 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:34.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.075+0000 7f7a89169700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7a841023f0 0x7f7a8419dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:34.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.075+0000 7f7a89169700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a84100860 0x7f7a84196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:34.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.075+0000 7f7a89169700 1 --2- 172.21.15.3:0/2866791918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a84101a50 0x7f7a84196bb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:34.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.075+0000 7f7a89169700 1 -- 172.21.15.3:0/2866791918 >> 172.21.15.3:0/2866791918 conn(0x7f7a840fbdd0 msgr2=0x7f7a84104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:34.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.076+0000 7f7a89169700 1 -- 172.21.15.3:0/2866791918 shutdown_connections 2023-07-20T01:42:34.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.077+0000 7f7a89169700 1 -- 172.21.15.3:0/2866791918 wait complete. 2023-07-20T01:42:34.044 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2023-07-20T01:42:34.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-20T01:42:34.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3059324376' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T01:42:34.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3059324376' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T01:42:34.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T01:42:34.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:34.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:42:34.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:33 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:42:34.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3059324376' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T01:42:34.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3059324376' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T01:42:34.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T01:42:34.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:34.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:42:34.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:33 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:42:34.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.226+0000 7f70c53c6700 1 -- 172.21.15.3:0/3827962240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70c01023b0 msgr2=0x7f70c0108c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:34.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.226+0000 7f70c53c6700 1 --2- 172.21.15.3:0/3827962240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70c01023b0 0x7f70c0108c60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f70b800b0d0 tx=0x7f70b800b3e0 comp rx=0 tx=0).stop 2023-07-20T01:42:34.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.226+0000 7f70c53c6700 1 -- 172.21.15.3:0/3827962240 shutdown_connections 2023-07-20T01:42:34.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.226+0000 7f70c53c6700 1 --2- 172.21.15.3:0/3827962240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70c01023b0 0x7f70c0108c60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:34.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.226+0000 7f70c53c6700 1 --2- 172.21.15.3:0/3827962240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c0100820 0x7f70c0100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:34.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.226+0000 7f70c53c6700 1 --2- 172.21.15.3:0/3827962240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c0101a10 0x7f70c0101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:34.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.226+0000 7f70c53c6700 1 -- 172.21.15.3:0/3827962240 >> 172.21.15.3:0/3827962240 conn(0x7f70c00fbd90 msgr2=0x7f70c00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:34.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.226+0000 7f70c53c6700 1 -- 172.21.15.3:0/3827962240 shutdown_connections 2023-07-20T01:42:34.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.226+0000 7f70c53c6700 1 -- 172.21.15.3:0/3827962240 wait complete. 2023-07-20T01:42:34.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70c53c6700 1 Processor -- start 2023-07-20T01:42:34.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70c53c6700 1 -- start start 2023-07-20T01:42:34.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70c53c6700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c0100820 0x7f70c01965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:34.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70c53c6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c0101a10 0x7f70c0196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:34.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70c53c6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70c01023b0 0x7f70c019db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:34.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70c53c6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c019e160 con 0x7f70c01023b0 2023-07-20T01:42:34.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70c53c6700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c019e2a0 con 0x7f70c0100820 2023-07-20T01:42:34.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70c53c6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c019e430 con 0x7f70c0101a10 2023-07-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70bf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70c01023b0 0x7f70c019db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70bf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70c01023b0 0x7f70c019db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42754/0 (socket says 172.21.15.3:42754) 2023-07-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70bf7fe700 1 -- 172.21.15.3:0/846028179 learned_addr learned my addr 172.21.15.3:0/846028179 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70be7fc700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c0101a10 0x7f70c0196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.227+0000 7f70beffd700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c0100820 0x7f70c01965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.228+0000 7f70bf7fe700 1 -- 172.21.15.3:0/846028179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c0101a10 msgr2=0x7f70c0196b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.228+0000 7f70bf7fe700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c0101a10 0x7f70c0196b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.228+0000 7f70bf7fe700 1 -- 172.21.15.3:0/846028179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c0100820 msgr2=0x7f70c01965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.228+0000 7f70bf7fe700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c0100820 0x7f70c01965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.228+0000 7f70bf7fe700 1 -- 172.21.15.3:0/846028179 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b8009d00 con 0x7f70c01023b0 2023-07-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.228+0000 7f70bf7fe700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70c01023b0 0x7f70c019db90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f70b8008ec0 tx=0x7f70b8008ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.228+0000 7f70b7fff700 1 -- 172.21.15.3:0/846028179 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b800e040 con 0x7f70c01023b0 2023-07-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.228+0000 7f70b7fff700 1 -- 172.21.15.3:0/846028179 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70b8005070 con 0x7f70c01023b0 2023-07-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.228+0000 7f70c53c6700 1 -- 172.21.15.3:0/846028179 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c00729a0 con 0x7f70c01023b0 2023-07-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.228+0000 7f70b7fff700 1 -- 172.21.15.3:0/846028179 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b801b6c0 con 0x7f70c01023b0 2023-07-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.228+0000 7f70c53c6700 1 -- 172.21.15.3:0/846028179 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c0072e50 con 0x7f70c01023b0 2023-07-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.229+0000 7f70c53c6700 1 -- 172.21.15.3:0/846028179 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c0107d10 con 0x7f70c01023b0 2023-07-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.230+0000 7f70b7fff700 1 -- 172.21.15.3:0/846028179 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f70b80051e0 con 0x7f70c01023b0 2023-07-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.230+0000 7f70b7fff700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f70ac0a1150 0x7f70ac0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.231+0000 7f70beffd700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f70ac0a1150 0x7f70ac0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.231+0000 7f70b7fff700 1 -- 172.21.15.3:0/846028179 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f70b80cdbd0 con 0x7f70c01023b0 2023-07-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.231+0000 7f70beffd700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f70ac0a1150 0x7f70ac0a3610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f70c0101870 tx=0x7f70b0012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.234+0000 7f70b7fff700 1 -- 172.21.15.3:0/846028179 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f70b8097560 con 0x7f70c01023b0 2023-07-20T01:42:34.251 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3059324376' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T01:42:34.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3059324376' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T01:42:34.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T01:42:34.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:34.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:42:34.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:34.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:33 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:42:34.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:34.491+0000 7f70c53c6700 1 -- 172.21.15.3:0/846028179 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f70c002c970 con 0x7f70c01023b0 2023-07-20T01:42:35.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:34 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2866791918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T01:42:35.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:34 smithi086 ceph-mon[98138]: pgmap v246: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:35.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:34 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2866791918' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T01:42:35.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:34 smithi086 ceph-mon[98138]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T01:42:35.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:34 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/846028179' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T01:42:35.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:34 smithi086 ceph-mon[98138]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-07-20T01:42:35.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.144+0000 7f70b7fff700 1 -- 172.21.15.3:0/846028179 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f70b809ab80 con 0x7f70c01023b0 2023-07-20T01:42:35.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.149+0000 7f70c53c6700 1 -- 172.21.15.3:0/846028179 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f70ac0a1150 msgr2=0x7f70ac0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.149+0000 7f70c53c6700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f70ac0a1150 0x7f70ac0a3610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f70c0101870 tx=0x7f70b0012040 comp rx=0 tx=0).stop 2023-07-20T01:42:35.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.149+0000 7f70c53c6700 1 -- 172.21.15.3:0/846028179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70c01023b0 msgr2=0x7f70c019db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.149+0000 7f70c53c6700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70c01023b0 0x7f70c019db90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f70b8008ec0 tx=0x7f70b8008ef0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.150+0000 7f70c53c6700 1 -- 172.21.15.3:0/846028179 shutdown_connections 2023-07-20T01:42:35.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.150+0000 7f70c53c6700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70c01023b0 0x7f70c019db90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.150+0000 7f70c53c6700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f70ac0a1150 0x7f70ac0a3610 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.150+0000 7f70c53c6700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70c0100820 0x7f70c01965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.150+0000 7f70c53c6700 1 --2- 172.21.15.3:0/846028179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c0101a10 0x7f70c0196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.150+0000 7f70c53c6700 1 -- 172.21.15.3:0/846028179 >> 172.21.15.3:0/846028179 conn(0x7f70c00fbd90 msgr2=0x7f70c0104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:35.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.150+0000 7f70c53c6700 1 -- 172.21.15.3:0/846028179 shutdown_connections 2023-07-20T01:42:35.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.150+0000 7f70c53c6700 1 -- 172.21.15.3:0/846028179 wait complete. 2023-07-20T01:42:35.119 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 2 and data pool 3 2023-07-20T01:42:35.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:34 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2866791918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T01:42:35.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:34 smithi003 ceph-mon[96422]: pgmap v246: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:35.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:34 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2866791918' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T01:42:35.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:34 smithi003 ceph-mon[96422]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T01:42:35.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:34 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/846028179' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T01:42:35.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:34 smithi003 ceph-mon[96422]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-07-20T01:42:35.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T01:42:35.251 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:34 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2866791918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T01:42:35.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:34 smithi043 ceph-mon[97823]: pgmap v246: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:35.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:34 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2866791918' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T01:42:35.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:34 smithi043 ceph-mon[97823]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T01:42:35.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:34 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/846028179' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T01:42:35.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:34 smithi043 ceph-mon[97823]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-07-20T01:42:35.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.342+0000 7faf8503e700 1 -- 172.21.15.3:0/2266731028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf801012d0 msgr2=0x7faf801036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.342+0000 7faf8503e700 1 --2- 172.21.15.3:0/2266731028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf801012d0 0x7faf801036c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faf70009990 tx=0x7faf70009ca0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.342+0000 7faf8503e700 1 -- 172.21.15.3:0/2266731028 shutdown_connections 2023-07-20T01:42:35.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.342+0000 7faf8503e700 1 --2- 172.21.15.3:0/2266731028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf80106530 0x7faf801089a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.342+0000 7faf8503e700 1 --2- 172.21.15.3:0/2266731028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf801012d0 0x7faf801036c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.342+0000 7faf8503e700 1 --2- 172.21.15.3:0/2266731028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf80103c00 0x7faf80105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.342+0000 7faf8503e700 1 -- 172.21.15.3:0/2266731028 >> 172.21.15.3:0/2266731028 conn(0x7faf800faca0 msgr2=0x7faf800fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:35.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.342+0000 7faf8503e700 1 -- 172.21.15.3:0/2266731028 shutdown_connections 2023-07-20T01:42:35.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.342+0000 7faf8503e700 1 -- 172.21.15.3:0/2266731028 wait complete. 2023-07-20T01:42:35.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf8503e700 1 Processor -- start 2023-07-20T01:42:35.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf8503e700 1 -- start start 2023-07-20T01:42:35.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf8503e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf801012d0 0x7faf801964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:35.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf8503e700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf80103c00 0x7faf80196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:35.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf8503e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf80106530 0x7faf8019daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:35.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf8503e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8019e070 con 0x7faf80106530 2023-07-20T01:42:35.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf8503e700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8019e1b0 con 0x7faf80103c00 2023-07-20T01:42:35.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf8503e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8019e340 con 0x7faf801012d0 2023-07-20T01:42:35.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf7f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf80106530 0x7faf8019daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:35.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf7ed9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf801012d0 0x7faf801964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:35.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf7f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf80106530 0x7faf8019daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42772/0 (socket says 172.21.15.3:42772) 2023-07-20T01:42:35.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf7e59c700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf80103c00 0x7faf80196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:35.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.343+0000 7faf7f59e700 1 -- 172.21.15.3:0/2593702739 learned_addr learned my addr 172.21.15.3:0/2593702739 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:35.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf7f59e700 1 -- 172.21.15.3:0/2593702739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf801012d0 msgr2=0x7faf801964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf7f59e700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf801012d0 0x7faf801964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf7f59e700 1 -- 172.21.15.3:0/2593702739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf80103c00 msgr2=0x7faf80196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf7f59e700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf80103c00 0x7faf80196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf7f59e700 1 -- 172.21.15.3:0/2593702739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf700095f0 con 0x7faf80106530 2023-07-20T01:42:35.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf7ed9d700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf801012d0 0x7faf801964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:42:35.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf7f59e700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf80106530 0x7faf8019daa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7faf7800eb80 tx=0x7faf7800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:35.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf67fff700 1 -- 172.21.15.3:0/2593702739 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78018070 con 0x7faf80106530 2023-07-20T01:42:35.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf67fff700 1 -- 172.21.15.3:0/2593702739 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf7800f040 con 0x7faf80106530 2023-07-20T01:42:35.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf8503e700 1 -- 172.21.15.3:0/2593702739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8019e5c0 con 0x7faf80106530 2023-07-20T01:42:35.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf67fff700 1 -- 172.21.15.3:0/2593702739 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7800cde0 con 0x7faf80106530 2023-07-20T01:42:35.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.344+0000 7faf8503e700 1 -- 172.21.15.3:0/2593702739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8019ea70 con 0x7faf80106530 2023-07-20T01:42:35.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.345+0000 7faf8503e700 1 -- 172.21.15.3:0/2593702739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf800fc820 con 0x7faf80106530 2023-07-20T01:42:35.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.347+0000 7faf67fff700 1 -- 172.21.15.3:0/2593702739 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7faf78003a50 con 0x7faf80106530 2023-07-20T01:42:35.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.348+0000 7faf67fff700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faf680a1150 0x7faf680a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:35.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.348+0000 7faf67fff700 1 -- 172.21.15.3:0/2593702739 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7faf780cc710 con 0x7faf80106530 2023-07-20T01:42:35.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.348+0000 7faf7ed9d700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faf680a1150 0x7faf680a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:35.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.348+0000 7faf7ed9d700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faf680a1150 0x7faf680a3610 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7faf7000b2e0 tx=0x7faf70005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:35.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.353+0000 7faf67fff700 1 -- 172.21.15.3:0/2593702739 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faf78095fb0 con 0x7faf80106530 2023-07-20T01:42:35.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.573+0000 7faf8503e700 1 -- 172.21.15.3:0/2593702739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf8002c970 con 0x7faf80106530 2023-07-20T01:42:35.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.574+0000 7faf67fff700 1 -- 172.21.15.3:0/2593702739 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+21950 (secure 0 0 0) 0x7faf780995d0 con 0x7faf80106530 2023-07-20T01:42:35.542 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:35.542 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":79,"fsid":"9b4fd2fc-269d-11ee-9b34-001a4aab830c","created":"2023-07-20T01:35:37.746283+0000","modified":"2023-07-20T01:42:34.492774+0000","last_up_change":"2023-07-20T01:41:53.551492+0000","last_in_change":"2023-07-20T01:41:35.988339+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T01:38:42.316631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T01:42:32.485461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T01:42:33.859667+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0073d11-a48f-415c-babd-5672ef2585b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6803","nonce":1627689053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6805","nonce":1627689053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6809","nonce":1627689053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6807","nonce":1627689053}]},"public_addr":"172.21.15.3:6803/1627689053","cluster_addr":"172.21.15.3:6805/1627689053","heartbeat_back_addr":"172.21.15.3:6809/1627689053","heartbeat_front_addr":"172.21.15.3:6807/1627689053","state":["exists","up"]},{"osd":1,"uuid":"91f60b3d-e08c-4a41-b480-c3d30348eacc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6801","nonce":3632963249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6803","nonce":3632963249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6807","nonce":3632963249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6805","nonce":3632963249}]},"public_addr":"172.21.15.43:6801/3632963249","cluster_addr":"172.21.15.43:6803/3632963249","heartbeat_back_addr":"172.21.15.43:6807/3632963249","heartbeat_front_addr":"172.21.15.43:6805/3632963249","state":["exists","up"]},{"osd":2,"uuid":"11808f4b-c50a-44f4-8581-3fc226653b39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6801","nonce":2141120676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6803","nonce":2141120676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6807","nonce":2141120676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6805","nonce":2141120676}]},"public_addr":"172.21.15.86:6801/2141120676","cluster_addr":"172.21.15.86:6803/2141120676","heartbeat_back_addr":"172.21.15.86:6807/2141120676","heartbeat_front_addr":"172.21.15.86:6805/2141120676","state":["exists","up"]},{"osd":3,"uuid":"45c2d3f6-9f5d-4510-9dad-6d95d044725f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6811","nonce":535233193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6813","nonce":535233193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6817","nonce":535233193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6815","nonce":535233193}]},"public_addr":"172.21.15.3:6811/535233193","cluster_addr":"172.21.15.3:6813/535233193","heartbeat_back_addr":"172.21.15.3:6817/535233193","heartbeat_front_addr":"172.21.15.3:6815/535233193","state":["exists","up"]},{"osd":4,"uuid":"7a35bbae-6f6c-4296-9459-f1b4d2de632f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6809","nonce":47431618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6811","nonce":47431618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6815","nonce":47431618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6813","nonce":47431618}]},"public_addr":"172.21.15.43:6809/47431618","cluster_addr":"172.21.15.43:6811/47431618","heartbeat_back_addr":"172.21.15.43:6815/47431618","heartbeat_front_addr":"172.21.15.43:6813/47431618","state":["exists","up"]},{"osd":5,"uuid":"34cccdff-563b-413c-b56c-08839b1534aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6809","nonce":1573392100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1573392100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1573392100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1573392100}]},"public_addr":"172.21.15.86:6809/1573392100","cluster_addr":"172.21.15.86:6811/1573392100","heartbeat_back_addr":"172.21.15.86:6815/1573392100","heartbeat_front_addr":"172.21.15.86:6813/1573392100","state":["exists","up"]},{"osd":6,"uuid":"e1c8f2b2-aed6-4277-b645-6a68e287ca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6819","nonce":1511361132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6821","nonce":1511361132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6825","nonce":1511361132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6823","nonce":1511361132}]},"public_addr":"172.21.15.3:6819/1511361132","cluster_addr":"172.21.15.3:6821/1511361132","heartbeat_back_addr":"172.21.15.3:6825/1511361132","heartbeat_front_addr":"172.21.15.3:6823/1511361132","state":["exists","up"]},{"osd":7,"uuid":"fe6f3c63-9935-43b0-858f-11fd5f91c9ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6817","nonce":3536213360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6819","nonce":3536213360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6823","nonce":3536213360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6821","nonce":3536213360}]},"public_addr":"172.21.15.43:6817/3536213360","cluster_addr":"172.21.15.43:6819/3536213360","heartbeat_back_addr":"172.21.15.43:6823/3536213360","heartbeat_front_addr":"172.21.15.43:6821/3536213360","state":["exists","up"]},{"osd":8,"uuid":"7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6817","nonce":2823536777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6819","nonce":2823536777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6823","nonce":2823536777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6821","nonce":2823536777}]},"public_addr":"172.21.15.86:6817/2823536777","cluster_addr":"172.21.15.86:6819/2823536777","heartbeat_back_addr":"172.21.15.86:6823/2823536777","heartbeat_front_addr":"172.21.15.86:6821/2823536777","state":["exists","up"]},{"osd":9,"uuid":"d356aa99-85dc-4006-9441-e4f7d23d594f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6827","nonce":272617110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6829","nonce":272617110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6833","nonce":272617110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6831","nonce":272617110}]},"public_addr":"172.21.15.3:6827/272617110","cluster_addr":"172.21.15.3:6829/272617110","heartbeat_back_addr":"172.21.15.3:6833/272617110","heartbeat_front_addr":"172.21.15.3:6831/272617110","state":["exists","up"]},{"osd":10,"uuid":"b8848e6f-3991-4755-8fd3-627c9ebcd823","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6825","nonce":161869649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6827","nonce":161869649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6831","nonce":161869649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6829","nonce":161869649}]},"public_addr":"172.21.15.43:6825/161869649","cluster_addr":"172.21.15.43:6827/161869649","heartbeat_back_addr":"172.21.15.43:6831/161869649","heartbeat_front_addr":"172.21.15.43:6829/161869649","state":["exists","up"]},{"osd":11,"uuid":"e0fb9b4d-8559-48c2-9b41-c61a83ca0a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6825","nonce":3393537360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6827","nonce":3393537360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6831","nonce":3393537360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6829","nonce":3393537360}]},"public_addr":"172.21.15.86:6825/3393537360","cluster_addr":"172.21.15.86:6827/3393537360","heartbeat_back_addr":"172.21.15.86:6831/3393537360","heartbeat_front_addr":"172.21.15.86:6829/3393537360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:37:59.181659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:38:20.326807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:38:40.402049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:02.811605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:23.889505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:44.425791+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:06.236232+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:27.676116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:47.846169+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:09.462761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:31.165679+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:51.723548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/942763937":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6801/596448098":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6800/596448098":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6800/417953684":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/1023086410":"2023-07-21T01:36:35.772246+0000","172.21.15.3:0/3900831138":"2023-07-21T01:36:35.772246+0000","172.21.15.3:0/2570287892":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/286795203":"2023-07-21T01:36:10.337325+0000","172.21.15.3:6801/417953684":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/2757371565":"2023-07-21T01:36:10.337325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T01:42:35.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 -- 172.21.15.3:0/2593702739 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faf680a1150 msgr2=0x7faf680a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faf680a1150 0x7faf680a3610 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7faf7000b2e0 tx=0x7faf70005fb0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 -- 172.21.15.3:0/2593702739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf80106530 msgr2=0x7faf8019daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf80106530 0x7faf8019daa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7faf7800eb80 tx=0x7faf7800ee90 comp rx=0 tx=0).stop 2023-07-20T01:42:35.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 -- 172.21.15.3:0/2593702739 shutdown_connections 2023-07-20T01:42:35.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf80106530 0x7faf8019daa0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf801012d0 0x7faf801964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7faf680a1150 0x7faf680a3610 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 --2- 172.21.15.3:0/2593702739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf80103c00 0x7faf80196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 -- 172.21.15.3:0/2593702739 >> 172.21.15.3:0/2593702739 conn(0x7faf800faca0 msgr2=0x7faf80101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:35.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 -- 172.21.15.3:0/2593702739 shutdown_connections 2023-07-20T01:42:35.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.576+0000 7faf8503e700 1 -- 172.21.15.3:0/2593702739 wait complete. 2023-07-20T01:42:35.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T01:42:35.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.721+0000 7fc5ac563700 1 -- 172.21.15.3:0/873139579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5a4100910 msgr2=0x7fc5a4100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.721+0000 7fc5ac563700 1 --2- 172.21.15.3:0/873139579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5a4100910 0x7fc5a4100d70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc5980099c0 tx=0x7fc598009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.721+0000 7fc5ac563700 1 -- 172.21.15.3:0/873139579 shutdown_connections 2023-07-20T01:42:35.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.721+0000 7fc5ac563700 1 --2- 172.21.15.3:0/873139579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5a41012b0 0x7fc5a4107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.721+0000 7fc5ac563700 1 --2- 172.21.15.3:0/873139579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a40ff720 0x7fc5a40ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.721+0000 7fc5ac563700 1 --2- 172.21.15.3:0/873139579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5a4100910 0x7fc5a4100d70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.721+0000 7fc5ac563700 1 -- 172.21.15.3:0/873139579 >> 172.21.15.3:0/873139579 conn(0x7fc5a40facb0 msgr2=0x7fc5a40fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:35.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.721+0000 7fc5ac563700 1 -- 172.21.15.3:0/873139579 shutdown_connections 2023-07-20T01:42:35.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.721+0000 7fc5ac563700 1 -- 172.21.15.3:0/873139579 wait complete. 2023-07-20T01:42:35.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5ac563700 1 Processor -- start 2023-07-20T01:42:35.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5ac563700 1 -- start start 2023-07-20T01:42:35.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5ac563700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5a40ff720 0x7fc5a4196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:35.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5ac563700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5a4100910 0x7fc5a4196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:35.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5ac563700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a41012b0 0x7fc5a419dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:35.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5ac563700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a419e080 con 0x7fc5a4100910 2023-07-20T01:42:35.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5ac563700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a419e1c0 con 0x7fc5a40ff720 2023-07-20T01:42:35.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5ac563700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a419e350 con 0x7fc5a41012b0 2023-07-20T01:42:35.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5a9afe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5a4100910 0x7fc5a4196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:35.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5a9afe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5a4100910 0x7fc5a4196a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42784/0 (socket says 172.21.15.3:42784) 2023-07-20T01:42:35.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5a9afe700 1 -- 172.21.15.3:0/334847131 learned_addr learned my addr 172.21.15.3:0/334847131 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:35.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5aa2ff700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5a40ff720 0x7fc5a4196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:35.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5aab00700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a41012b0 0x7fc5a419dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:35.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5a9afe700 1 -- 172.21.15.3:0/334847131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a41012b0 msgr2=0x7fc5a419dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5a9afe700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a41012b0 0x7fc5a419dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5a9afe700 1 -- 172.21.15.3:0/334847131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5a40ff720 msgr2=0x7fc5a4196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5a9afe700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5a40ff720 0x7fc5a4196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5a9afe700 1 -- 172.21.15.3:0/334847131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5980095f0 con 0x7fc5a4100910 2023-07-20T01:42:35.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5aa2ff700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5a40ff720 0x7fc5a4196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:35.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.722+0000 7fc5a9afe700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5a4100910 0x7fc5a4196a40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc598009990 tx=0x7fc59800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:35.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.723+0000 7fc5977fe700 1 -- 172.21.15.3:0/334847131 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59801d070 con 0x7fc5a4100910 2023-07-20T01:42:35.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.723+0000 7fc5977fe700 1 -- 172.21.15.3:0/334847131 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5980054f0 con 0x7fc5a4100910 2023-07-20T01:42:35.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.723+0000 7fc5ac563700 1 -- 172.21.15.3:0/334847131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a419e5d0 con 0x7fc5a4100910 2023-07-20T01:42:35.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.723+0000 7fc5977fe700 1 -- 172.21.15.3:0/334847131 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc598004c70 con 0x7fc5a4100910 2023-07-20T01:42:35.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.723+0000 7fc5ac563700 1 -- 172.21.15.3:0/334847131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a419e9f0 con 0x7fc5a4100910 2023-07-20T01:42:35.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.724+0000 7fc5ac563700 1 -- 172.21.15.3:0/334847131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a410af90 con 0x7fc5a4100910 2023-07-20T01:42:35.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.725+0000 7fc5977fe700 1 -- 172.21.15.3:0/334847131 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc59800fc30 con 0x7fc5a4100910 2023-07-20T01:42:35.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.725+0000 7fc5977fe700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fc5900a1150 0x7fc5900a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:35.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.726+0000 7fc5aa2ff700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fc5900a1150 0x7fc5900a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:35.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.726+0000 7fc5977fe700 1 -- 172.21.15.3:0/334847131 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc5980cdaf0 con 0x7fc5a4100910 2023-07-20T01:42:35.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.726+0000 7fc5aa2ff700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fc5900a1150 0x7fc5900a3610 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0005950 tx=0x7fc5a00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:35.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.729+0000 7fc5977fe700 1 -- 172.21.15.3:0/334847131 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc598097420 con 0x7fc5a4100910 2023-07-20T01:42:35.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.952+0000 7fc5ac563700 1 -- 172.21.15.3:0/334847131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5a419f030 con 0x7fc5a4100910 2023-07-20T01:42:35.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.953+0000 7fc5977fe700 1 -- 172.21.15.3:0/334847131 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+21937 (secure 0 0 0) 0x7fc59809aa40 con 0x7fc5a4100910 2023-07-20T01:42:35.921 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-07-20T01:42:35.922 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":80,"fsid":"9b4fd2fc-269d-11ee-9b34-001a4aab830c","created":"2023-07-20T01:35:37.746283+0000","modified":"2023-07-20T01:42:35.862724+0000","last_up_change":"2023-07-20T01:41:53.551492+0000","last_in_change":"2023-07-20T01:41:35.988339+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T01:38:42.316631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T01:42:32.485461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T01:42:33.859667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0073d11-a48f-415c-babd-5672ef2585b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6803","nonce":1627689053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6805","nonce":1627689053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6809","nonce":1627689053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1627689053},{"type":"v1","addr":"172.21.15.3:6807","nonce":1627689053}]},"public_addr":"172.21.15.3:6803/1627689053","cluster_addr":"172.21.15.3:6805/1627689053","heartbeat_back_addr":"172.21.15.3:6809/1627689053","heartbeat_front_addr":"172.21.15.3:6807/1627689053","state":["exists","up"]},{"osd":1,"uuid":"91f60b3d-e08c-4a41-b480-c3d30348eacc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6801","nonce":3632963249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6803","nonce":3632963249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6807","nonce":3632963249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3632963249},{"type":"v1","addr":"172.21.15.43:6805","nonce":3632963249}]},"public_addr":"172.21.15.43:6801/3632963249","cluster_addr":"172.21.15.43:6803/3632963249","heartbeat_back_addr":"172.21.15.43:6807/3632963249","heartbeat_front_addr":"172.21.15.43:6805/3632963249","state":["exists","up"]},{"osd":2,"uuid":"11808f4b-c50a-44f4-8581-3fc226653b39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6801","nonce":2141120676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6803","nonce":2141120676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6807","nonce":2141120676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2141120676},{"type":"v1","addr":"172.21.15.86:6805","nonce":2141120676}]},"public_addr":"172.21.15.86:6801/2141120676","cluster_addr":"172.21.15.86:6803/2141120676","heartbeat_back_addr":"172.21.15.86:6807/2141120676","heartbeat_front_addr":"172.21.15.86:6805/2141120676","state":["exists","up"]},{"osd":3,"uuid":"45c2d3f6-9f5d-4510-9dad-6d95d044725f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6811","nonce":535233193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6813","nonce":535233193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6817","nonce":535233193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":535233193},{"type":"v1","addr":"172.21.15.3:6815","nonce":535233193}]},"public_addr":"172.21.15.3:6811/535233193","cluster_addr":"172.21.15.3:6813/535233193","heartbeat_back_addr":"172.21.15.3:6817/535233193","heartbeat_front_addr":"172.21.15.3:6815/535233193","state":["exists","up"]},{"osd":4,"uuid":"7a35bbae-6f6c-4296-9459-f1b4d2de632f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6809","nonce":47431618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6811","nonce":47431618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6815","nonce":47431618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":47431618},{"type":"v1","addr":"172.21.15.43:6813","nonce":47431618}]},"public_addr":"172.21.15.43:6809/47431618","cluster_addr":"172.21.15.43:6811/47431618","heartbeat_back_addr":"172.21.15.43:6815/47431618","heartbeat_front_addr":"172.21.15.43:6813/47431618","state":["exists","up"]},{"osd":5,"uuid":"34cccdff-563b-413c-b56c-08839b1534aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6809","nonce":1573392100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1573392100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1573392100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1573392100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1573392100}]},"public_addr":"172.21.15.86:6809/1573392100","cluster_addr":"172.21.15.86:6811/1573392100","heartbeat_back_addr":"172.21.15.86:6815/1573392100","heartbeat_front_addr":"172.21.15.86:6813/1573392100","state":["exists","up"]},{"osd":6,"uuid":"e1c8f2b2-aed6-4277-b645-6a68e287ca2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6819","nonce":1511361132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6821","nonce":1511361132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6825","nonce":1511361132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1511361132},{"type":"v1","addr":"172.21.15.3:6823","nonce":1511361132}]},"public_addr":"172.21.15.3:6819/1511361132","cluster_addr":"172.21.15.3:6821/1511361132","heartbeat_back_addr":"172.21.15.3:6825/1511361132","heartbeat_front_addr":"172.21.15.3:6823/1511361132","state":["exists","up"]},{"osd":7,"uuid":"fe6f3c63-9935-43b0-858f-11fd5f91c9ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6817","nonce":3536213360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6819","nonce":3536213360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6823","nonce":3536213360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":3536213360},{"type":"v1","addr":"172.21.15.43:6821","nonce":3536213360}]},"public_addr":"172.21.15.43:6817/3536213360","cluster_addr":"172.21.15.43:6819/3536213360","heartbeat_back_addr":"172.21.15.43:6823/3536213360","heartbeat_front_addr":"172.21.15.43:6821/3536213360","state":["exists","up"]},{"osd":8,"uuid":"7a7e3aaf-628e-45ab-8e10-8fbec6c1dfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6817","nonce":2823536777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6819","nonce":2823536777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6823","nonce":2823536777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":2823536777},{"type":"v1","addr":"172.21.15.86:6821","nonce":2823536777}]},"public_addr":"172.21.15.86:6817/2823536777","cluster_addr":"172.21.15.86:6819/2823536777","heartbeat_back_addr":"172.21.15.86:6823/2823536777","heartbeat_front_addr":"172.21.15.86:6821/2823536777","state":["exists","up"]},{"osd":9,"uuid":"d356aa99-85dc-4006-9441-e4f7d23d594f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6827","nonce":272617110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6829","nonce":272617110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6833","nonce":272617110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":272617110},{"type":"v1","addr":"172.21.15.3:6831","nonce":272617110}]},"public_addr":"172.21.15.3:6827/272617110","cluster_addr":"172.21.15.3:6829/272617110","heartbeat_back_addr":"172.21.15.3:6833/272617110","heartbeat_front_addr":"172.21.15.3:6831/272617110","state":["exists","up"]},{"osd":10,"uuid":"b8848e6f-3991-4755-8fd3-627c9ebcd823","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6825","nonce":161869649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6827","nonce":161869649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6831","nonce":161869649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":161869649},{"type":"v1","addr":"172.21.15.43:6829","nonce":161869649}]},"public_addr":"172.21.15.43:6825/161869649","cluster_addr":"172.21.15.43:6827/161869649","heartbeat_back_addr":"172.21.15.43:6831/161869649","heartbeat_front_addr":"172.21.15.43:6829/161869649","state":["exists","up"]},{"osd":11,"uuid":"e0fb9b4d-8559-48c2-9b41-c61a83ca0a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6825","nonce":3393537360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6827","nonce":3393537360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6831","nonce":3393537360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":3393537360},{"type":"v1","addr":"172.21.15.86:6829","nonce":3393537360}]},"public_addr":"172.21.15.86:6825/3393537360","cluster_addr":"172.21.15.86:6827/3393537360","heartbeat_back_addr":"172.21.15.86:6831/3393537360","heartbeat_front_addr":"172.21.15.86:6829/3393537360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:37:59.181659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:38:20.326807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:38:40.402049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:02.811605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:23.889505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:39:44.425791+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:06.236232+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:27.676116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:40:47.846169+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:09.462761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:31.165679+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:41:51.723548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/942763937":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6801/596448098":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6800/596448098":"2023-07-21T01:36:35.772246+0000","172.21.15.3:6800/417953684":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/1023086410":"2023-07-21T01:36:35.772246+0000","172.21.15.3:0/3900831138":"2023-07-21T01:36:35.772246+0000","172.21.15.3:0/2570287892":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/286795203":"2023-07-21T01:36:10.337325+0000","172.21.15.3:6801/417953684":"2023-07-21T01:36:10.337325+0000","172.21.15.3:0/2757371565":"2023-07-21T01:36:10.337325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T01:42:35.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 -- 172.21.15.3:0/334847131 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fc5900a1150 msgr2=0x7fc5900a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fc5900a1150 0x7fc5900a3610 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0005950 tx=0x7fc5a00058e0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 -- 172.21.15.3:0/334847131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5a4100910 msgr2=0x7fc5a4196a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5a4100910 0x7fc5a4196a40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc598009990 tx=0x7fc59800f740 comp rx=0 tx=0).stop 2023-07-20T01:42:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 -- 172.21.15.3:0/334847131 shutdown_connections 2023-07-20T01:42:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a41012b0 0x7fc5a419dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fc5900a1150 0x7fc5900a3610 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5a40ff720 0x7fc5a4196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 --2- 172.21.15.3:0/334847131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5a4100910 0x7fc5a4196a40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 -- 172.21.15.3:0/334847131 >> 172.21.15.3:0/334847131 conn(0x7fc5a40facb0 msgr2=0x7fc5a4067a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 -- 172.21.15.3:0/334847131 shutdown_connections 2023-07-20T01:42:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:35.955+0000 7fc5ac563700 1 -- 172.21.15.3:0/334847131 wait complete. 2023-07-20T01:42:35.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-20T01:42:36.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.102+0000 7f8d314aa700 1 -- 172.21.15.3:0/2821088414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d2c103c00 msgr2=0x7f8d2c105ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.102+0000 7f8d314aa700 1 --2- 172.21.15.3:0/2821088414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d2c103c00 0x7f8d2c105ff0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009990 tx=0x7f8d20009ca0 comp rx=0 tx=0).stop 2023-07-20T01:42:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.102+0000 7f8d314aa700 1 -- 172.21.15.3:0/2821088414 shutdown_connections 2023-07-20T01:42:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.102+0000 7f8d314aa700 1 --2- 172.21.15.3:0/2821088414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d2c106530 0x7f8d2c1089a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.102+0000 7f8d314aa700 1 --2- 172.21.15.3:0/2821088414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d2c1012d0 0x7f8d2c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.102+0000 7f8d314aa700 1 --2- 172.21.15.3:0/2821088414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d2c103c00 0x7f8d2c105ff0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.102+0000 7f8d314aa700 1 -- 172.21.15.3:0/2821088414 >> 172.21.15.3:0/2821088414 conn(0x7f8d2c0face0 msgr2=0x7f8d2c0fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.102+0000 7f8d314aa700 1 -- 172.21.15.3:0/2821088414 shutdown_connections 2023-07-20T01:42:36.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.102+0000 7f8d314aa700 1 -- 172.21.15.3:0/2821088414 wait complete. 2023-07-20T01:42:36.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.102+0000 7f8d314aa700 1 Processor -- start 2023-07-20T01:42:36.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d314aa700 1 -- start start 2023-07-20T01:42:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d314aa700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d2c1012d0 0x7f8d2c196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d314aa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d2c103c00 0x7f8d2c196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d314aa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d2c106530 0x7f8d2c19dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d314aa700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c19e090 con 0x7f8d2c103c00 2023-07-20T01:42:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d314aa700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c19e1d0 con 0x7f8d2c1012d0 2023-07-20T01:42:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d314aa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c19e360 con 0x7f8d2c106530 2023-07-20T01:42:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d2a7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d2c103c00 0x7f8d2c196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d2a7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d2c103c00 0x7f8d2c196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42798/0 (socket says 172.21.15.3:42798) 2023-07-20T01:42:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d2a7fc700 1 -- 172.21.15.3:0/1480661515 learned_addr learned my addr 172.21.15.3:0/1480661515 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d2affd700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d2c1012d0 0x7f8d2c196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d2b7fe700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d2c106530 0x7f8d2c19dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d2a7fc700 1 -- 172.21.15.3:0/1480661515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d2c106530 msgr2=0x7f8d2c19dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d2a7fc700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d2c106530 0x7f8d2c19dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d2a7fc700 1 -- 172.21.15.3:0/1480661515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d2c1012d0 msgr2=0x7f8d2c196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d2a7fc700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d2c1012d0 0x7f8d2c196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d2a7fc700 1 -- 172.21.15.3:0/1480661515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200095f0 con 0x7f8d2c103c00 2023-07-20T01:42:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.103+0000 7f8d2affd700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d2c1012d0 0x7f8d2c196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:42:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.104+0000 7f8d2a7fc700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d2c103c00 0x7f8d2c196a50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8d200059d0 tx=0x7f8d200047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.104+0000 7f8d13fff700 1 -- 172.21.15.3:0/1480661515 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2001d070 con 0x7f8d2c103c00 2023-07-20T01:42:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.104+0000 7f8d314aa700 1 -- 172.21.15.3:0/1480661515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c19e5e0 con 0x7f8d2c103c00 2023-07-20T01:42:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.104+0000 7f8d13fff700 1 -- 172.21.15.3:0/1480661515 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d2000fb90 con 0x7f8d2c103c00 2023-07-20T01:42:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.104+0000 7f8d13fff700 1 -- 172.21.15.3:0/1480661515 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20017650 con 0x7f8d2c103c00 2023-07-20T01:42:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.104+0000 7f8d314aa700 1 -- 172.21.15.3:0/1480661515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c19ea00 con 0x7f8d2c103c00 2023-07-20T01:42:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.105+0000 7f8d314aa700 1 -- 172.21.15.3:0/1480661515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c0fc860 con 0x7f8d2c103c00 2023-07-20T01:42:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.106+0000 7f8d13fff700 1 -- 172.21.15.3:0/1480661515 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8d2000fd00 con 0x7f8d2c103c00 2023-07-20T01:42:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.107+0000 7f8d13fff700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8d140a1150 0x7f8d140a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.107+0000 7f8d2affd700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8d140a1150 0x7f8d140a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.107+0000 7f8d13fff700 1 -- 172.21.15.3:0/1480661515 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8d200ce330 con 0x7f8d2c103c00 2023-07-20T01:42:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.107+0000 7f8d2affd700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8d140a1150 0x7f8d140a3610 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c011490 tx=0x7f8d1c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.110+0000 7f8d13fff700 1 -- 172.21.15.3:0/1480661515 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8d20097c60 con 0x7f8d2c103c00 2023-07-20T01:42:36.101 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:35 smithi086 ceph-mon[98138]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T01:42:36.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:35 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/846028179' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T01:42:36.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:35 smithi086 ceph-mon[98138]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-07-20T01:42:36.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:35 smithi086 ceph-mon[98138]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-07-20T01:42:36.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:35 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/2593702739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:36.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:35 smithi003 ceph-mon[96422]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T01:42:36.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:35 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/846028179' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T01:42:36.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:35 smithi003 ceph-mon[96422]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-07-20T01:42:36.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:35 smithi003 ceph-mon[96422]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-07-20T01:42:36.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:35 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/2593702739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:36.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:35 smithi043 ceph-mon[97823]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T01:42:36.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:35 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/846028179' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T01:42:36.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:35 smithi043 ceph-mon[97823]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-07-20T01:42:36.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:35 smithi043 ceph-mon[97823]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-07-20T01:42:36.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:35 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/2593702739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:36.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:36.367+0000 7f8d314aa700 1 -- 172.21.15.3:0/1480661515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8d2c02c970 con 0x7f8d2c103c00 2023-07-20T01:42:37.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: pgmap v249: 97 pgs: 13 creating+peering, 18 active+clean, 66 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:37.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T01:42:37.102 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/334847131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:37.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:37.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:42:37.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.86:6838/1279793907,v1:172.21.15.86:6839/1279793907] up:active 2023-07-20T01:42:37.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:37.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:37.104 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:36 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/1480661515' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T01:42:37.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.151+0000 7f8d13fff700 1 -- 172.21.15.3:0/1480661515 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8d2009b280 con 0x7f8d2c103c00 2023-07-20T01:42:37.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 -- 172.21.15.3:0/1480661515 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8d140a1150 msgr2=0x7f8d140a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:37.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8d140a1150 0x7f8d140a3610 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c011490 tx=0x7f8d1c009260 comp rx=0 tx=0).stop 2023-07-20T01:42:37.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 -- 172.21.15.3:0/1480661515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d2c103c00 msgr2=0x7f8d2c196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:37.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d2c103c00 0x7f8d2c196a50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8d200059d0 tx=0x7f8d200047a0 comp rx=0 tx=0).stop 2023-07-20T01:42:37.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 -- 172.21.15.3:0/1480661515 shutdown_connections 2023-07-20T01:42:37.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d2c106530 0x7f8d2c19dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:37.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f8d140a1150 0x7f8d140a3610 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:37.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d2c1012d0 0x7f8d2c196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:37.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 --2- 172.21.15.3:0/1480661515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d2c103c00 0x7f8d2c196a50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:37.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 -- 172.21.15.3:0/1480661515 >> 172.21.15.3:0/1480661515 conn(0x7f8d2c0face0 msgr2=0x7f8d2c101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:37.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 -- 172.21.15.3:0/1480661515 shutdown_connections 2023-07-20T01:42:37.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.154+0000 7f8d314aa700 1 -- 172.21.15.3:0/1480661515 wait complete. 2023-07-20T01:42:37.136 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: pgmap v249: 97 pgs: 13 creating+peering, 18 active+clean, 66 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:37.136 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T01:42:37.136 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/334847131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:37.136 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:37.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:42:37.138 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.138 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.86:6838/1279793907,v1:172.21.15.86:6839/1279793907] up:active 2023-07-20T01:42:37.138 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:37.138 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:37.138 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:36 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/1480661515' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T01:42:37.140 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2023-07-20T01:42:37.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-07-20T01:42:37.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: pgmap v249: 97 pgs: 13 creating+peering, 18 active+clean, 66 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:42:37.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T01:42:37.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/334847131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:42:37.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.253 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:42:37.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:42:37.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: from='mgr.14152 172.21.15.3:0/2723070826' entity='mgr.x' 2023-07-20T01:42:37.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.86:6838/1279793907,v1:172.21.15.86:6839/1279793907] up:active 2023-07-20T01:42:37.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:37.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:37.254 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:36 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/1480661515' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T01:42:37.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.309+0000 7f548cdac700 1 -- 172.21.15.3:0/2898978602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5488100930 msgr2=0x7f5488100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:37.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.309+0000 7f548cdac700 1 --2- 172.21.15.3:0/2898978602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5488100930 0x7f5488100d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f54700099c0 tx=0x7f5470009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:37.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.309+0000 7f548cdac700 1 -- 172.21.15.3:0/2898978602 shutdown_connections 2023-07-20T01:42:37.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.309+0000 7f548cdac700 1 --2- 172.21.15.3:0/2898978602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54881012d0 0x7f5488107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:37.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.309+0000 7f548cdac700 1 --2- 172.21.15.3:0/2898978602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54880ff740 0x7f54880ffb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:37.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.309+0000 7f548cdac700 1 --2- 172.21.15.3:0/2898978602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5488100930 0x7f5488100d90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:37.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.309+0000 7f548cdac700 1 -- 172.21.15.3:0/2898978602 >> 172.21.15.3:0/2898978602 conn(0x7f54880facf0 msgr2=0x7f54880fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:37.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.309+0000 7f548cdac700 1 -- 172.21.15.3:0/2898978602 shutdown_connections 2023-07-20T01:42:37.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.309+0000 7f548cdac700 1 -- 172.21.15.3:0/2898978602 wait complete. 2023-07-20T01:42:37.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f548cdac700 1 Processor -- start 2023-07-20T01:42:37.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f548cdac700 1 -- start start 2023-07-20T01:42:37.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f548cdac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54880ff740 0x7f5488068610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:37.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f548cdac700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5488100930 0x7f548819b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:37.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f548cdac700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54881012d0 0x7f548819d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:37.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f548cdac700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548819dfe0 con 0x7f54880ff740 2023-07-20T01:42:37.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f548cdac700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548819e150 con 0x7f54881012d0 2023-07-20T01:42:37.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f548cdac700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548819e310 con 0x7f5488100930 2023-07-20T01:42:37.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f548659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54880ff740 0x7f5488068610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:37.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f548659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54880ff740 0x7f5488068610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42810/0 (socket says 172.21.15.3:42810) 2023-07-20T01:42:37.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f548659c700 1 -- 172.21.15.3:0/717047014 learned_addr learned my addr 172.21.15.3:0/717047014 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:37.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.310+0000 7f5485d9b700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5488100930 0x7f548819b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:37.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f5486d9d700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54881012d0 0x7f548819d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:37.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f548659c700 1 -- 172.21.15.3:0/717047014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5488100930 msgr2=0x7f548819b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:37.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f548659c700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5488100930 0x7f548819b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:37.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f548659c700 1 -- 172.21.15.3:0/717047014 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54881012d0 msgr2=0x7f548819d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:37.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f548659c700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54881012d0 0x7f548819d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:37.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f548659c700 1 -- 172.21.15.3:0/717047014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54700095f0 con 0x7f54880ff740 2023-07-20T01:42:37.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f5486d9d700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54881012d0 0x7f548819d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:37.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f548659c700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54880ff740 0x7f5488068610 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f547800d980 tx=0x7f547800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:37.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f547f7fe700 1 -- 172.21.15.3:0/717047014 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478004020 con 0x7f54880ff740 2023-07-20T01:42:37.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f547f7fe700 1 -- 172.21.15.3:0/717047014 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5478009730 con 0x7f54880ff740 2023-07-20T01:42:37.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f548cdac700 1 -- 172.21.15.3:0/717047014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548819e5f0 con 0x7f54880ff740 2023-07-20T01:42:37.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f547f7fe700 1 -- 172.21.15.3:0/717047014 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478003a80 con 0x7f54880ff740 2023-07-20T01:42:37.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.311+0000 7f548cdac700 1 -- 172.21.15.3:0/717047014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548819eb00 con 0x7f54880ff740 2023-07-20T01:42:37.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.312+0000 7f548cdac700 1 -- 172.21.15.3:0/717047014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5488106670 con 0x7f54880ff740 2023-07-20T01:42:37.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.313+0000 7f547f7fe700 1 -- 172.21.15.3:0/717047014 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f54780098a0 con 0x7f54880ff740 2023-07-20T01:42:37.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.314+0000 7f547f7fe700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54740a1150 0x7f54740a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:37.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.314+0000 7f5485d9b700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54740a1150 0x7f54740a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:37.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.314+0000 7f547f7fe700 1 -- 172.21.15.3:0/717047014 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f54780cc8a0 con 0x7f54880ff740 2023-07-20T01:42:37.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.314+0000 7f5485d9b700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54740a1150 0x7f54740a3610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f547000b580 tx=0x7f5470005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:37.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.317+0000 7f547f7fe700 1 -- 172.21.15.3:0/717047014 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f54780960a0 con 0x7f54880ff740 2023-07-20T01:42:37.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:37.568+0000 7f548cdac700 1 -- 172.21.15.3:0/717047014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f548802cc00 con 0x7f54880ff740 2023-07-20T01:42:38.351 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:42:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-i[124445]: 2023-07-20T01:42:37.948+0000 7f1001dc7700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-20T01:42:38.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:38 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/1480661515' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T01:42:38.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:38 smithi086 ceph-mon[98138]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:38.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:38 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/717047014' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T01:42:38.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:38 smithi086 ceph-mon[98138]: pgmap v251: 97 pgs: 13 creating+peering, 69 active+clean, 15 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 3 op/s 2023-07-20T01:42:38.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:38 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/1480661515' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T01:42:38.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:38 smithi003 ceph-mon[96422]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:38.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:38 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/717047014' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T01:42:38.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:38 smithi003 ceph-mon[96422]: pgmap v251: 97 pgs: 13 creating+peering, 69 active+clean, 15 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 3 op/s 2023-07-20T01:42:38.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.495+0000 7f547f7fe700 1 -- 172.21.15.3:0/717047014 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f54780996c0 con 0x7f54880ff740 2023-07-20T01:42:38.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 -- 172.21.15.3:0/717047014 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54740a1150 msgr2=0x7f54740a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:38.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54740a1150 0x7f54740a3610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f547000b580 tx=0x7f5470005fb0 comp rx=0 tx=0).stop 2023-07-20T01:42:38.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 -- 172.21.15.3:0/717047014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54880ff740 msgr2=0x7f5488068610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:38.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54880ff740 0x7f5488068610 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f547800d980 tx=0x7f547800dc90 comp rx=0 tx=0).stop 2023-07-20T01:42:38.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 -- 172.21.15.3:0/717047014 shutdown_connections 2023-07-20T01:42:38.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f54740a1150 0x7f54740a3610 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:38.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54881012d0 0x7f548819d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:38.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54880ff740 0x7f5488068610 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:38.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 --2- 172.21.15.3:0/717047014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5488100930 0x7f548819b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:38.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 -- 172.21.15.3:0/717047014 >> 172.21.15.3:0/717047014 conn(0x7f54880facf0 msgr2=0x7f54880fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:38.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 -- 172.21.15.3:0/717047014 shutdown_connections 2023-07-20T01:42:38.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.498+0000 7f548cdac700 1 -- 172.21.15.3:0/717047014 wait complete. 2023-07-20T01:42:38.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-20T01:42:38.501 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:38 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/1480661515' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T01:42:38.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:38 smithi043 ceph-mon[97823]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:38.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:38 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/717047014' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T01:42:38.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:38 smithi043 ceph-mon[97823]: pgmap v251: 97 pgs: 13 creating+peering, 69 active+clean, 15 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 3 op/s 2023-07-20T01:42:38.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.650+0000 7fb154758700 1 -- 172.21.15.3:0/2061827013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14c100380 msgr2=0x7fb14c107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:38.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.650+0000 7fb154758700 1 --2- 172.21.15.3:0/2061827013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14c100380 0x7fb14c107c40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb14400b0a0 tx=0x7fb14400b3b0 comp rx=0 tx=0).stop 2023-07-20T01:42:38.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.650+0000 7fb154758700 1 -- 172.21.15.3:0/2061827013 shutdown_connections 2023-07-20T01:42:38.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.650+0000 7fb154758700 1 --2- 172.21.15.3:0/2061827013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14c100380 0x7fb14c107c40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:38.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.650+0000 7fb154758700 1 --2- 172.21.15.3:0/2061827013 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb14c100d40 0x7fb14c101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:38.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.650+0000 7fb154758700 1 --2- 172.21.15.3:0/2061827013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb14c0ff9e0 0x7fb14c0ffe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:38.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.650+0000 7fb154758700 1 -- 172.21.15.3:0/2061827013 >> 172.21.15.3:0/2061827013 conn(0x7fb14c0fa450 msgr2=0x7fb14c0fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:38.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.650+0000 7fb154758700 1 -- 172.21.15.3:0/2061827013 shutdown_connections 2023-07-20T01:42:38.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.650+0000 7fb154758700 1 -- 172.21.15.3:0/2061827013 wait complete. 2023-07-20T01:42:38.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb154758700 1 Processor -- start 2023-07-20T01:42:38.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb154758700 1 -- start start 2023-07-20T01:42:38.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb154758700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb14c0ff9e0 0x7fb14c0ff440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:38.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb154758700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb14c100380 0x7fb14c19d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:38.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb154758700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14c100d40 0x7fb14c19fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:38.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb154758700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c1a03d0 con 0x7fb14c100d40 2023-07-20T01:42:38.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb154758700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c1a0540 con 0x7fb14c0ff9e0 2023-07-20T01:42:38.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb154758700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c1a0700 con 0x7fb14c100380 2023-07-20T01:42:38.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb152cf5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14c100d40 0x7fb14c19fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:38.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb152cf5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14c100d40 0x7fb14c19fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42826/0 (socket says 172.21.15.3:42826) 2023-07-20T01:42:38.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb152cf5700 1 -- 172.21.15.3:0/4058343491 learned_addr learned my addr 172.21.15.3:0/4058343491 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:38.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb1524f4700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb14c0ff9e0 0x7fb14c0ff440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:38.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb151cf3700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb14c100380 0x7fb14c19d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:38.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb152cf5700 1 -- 172.21.15.3:0/4058343491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb14c100380 msgr2=0x7fb14c19d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:38.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb152cf5700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb14c100380 0x7fb14c19d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:38.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb152cf5700 1 -- 172.21.15.3:0/4058343491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb14c0ff9e0 msgr2=0x7fb14c0ff440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:38.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb152cf5700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb14c0ff9e0 0x7fb14c0ff440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:38.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb152cf5700 1 -- 172.21.15.3:0/4058343491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb144009d00 con 0x7fb14c100d40 2023-07-20T01:42:38.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.651+0000 7fb152cf5700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14c100d40 0x7fb14c19fd20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb14400b780 tx=0x7fb14400bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:38.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.652+0000 7fb1437fe700 1 -- 172.21.15.3:0/4058343491 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14400e040 con 0x7fb14c100d40 2023-07-20T01:42:38.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.652+0000 7fb1437fe700 1 -- 172.21.15.3:0/4058343491 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb144004c50 con 0x7fb14c100d40 2023-07-20T01:42:38.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.652+0000 7fb154758700 1 -- 172.21.15.3:0/4058343491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14c1a0980 con 0x7fb14c100d40 2023-07-20T01:42:38.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.652+0000 7fb1437fe700 1 -- 172.21.15.3:0/4058343491 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb144005280 con 0x7fb14c100d40 2023-07-20T01:42:38.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.652+0000 7fb154758700 1 -- 172.21.15.3:0/4058343491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14c1a0e50 con 0x7fb14c100d40 2023-07-20T01:42:38.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.653+0000 7fb154758700 1 -- 172.21.15.3:0/4058343491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14c106d90 con 0x7fb14c100d40 2023-07-20T01:42:38.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.654+0000 7fb1437fe700 1 -- 172.21.15.3:0/4058343491 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fb14401d470 con 0x7fb14c100d40 2023-07-20T01:42:38.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.654+0000 7fb1437fe700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb1380a11d0 0x7fb1380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:38.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.654+0000 7fb1437fe700 1 -- 172.21.15.3:0/4058343491 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb1440ce3e0 con 0x7fb14c100d40 2023-07-20T01:42:38.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.654+0000 7fb1524f4700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb1380a11d0 0x7fb1380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:38.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.655+0000 7fb1524f4700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb1380a11d0 0x7fb1380a3690 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb14c067800 tx=0x7fb13c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:38.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.658+0000 7fb1437fe700 1 -- 172.21.15.3:0/4058343491 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb144097d10 con 0x7fb14c100d40 2023-07-20T01:42:38.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:38.905+0000 7fb154758700 1 -- 172.21.15.3:0/4058343491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fb14c02c970 con 0x7fb14c100d40 2023-07-20T01:42:39.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:39 smithi003 ceph-mon[96422]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T01:42:39.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:39 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/717047014' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T01:42:39.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:39 smithi003 ceph-mon[96422]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T01:42:39.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:39 smithi003 ceph-mon[96422]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:39.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:39 smithi003 ceph-mon[96422]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2023-07-20T01:42:39.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:39 smithi003 ceph-mon[96422]: daemon mds.b is now active in filesystem cephfs as rank 1 2023-07-20T01:42:39.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:39 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/4058343491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T01:42:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.503+0000 7fb1437fe700 1 -- 172.21.15.3:0/4058343491 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7fb14409b330 con 0x7fb14c100d40 2023-07-20T01:42:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.505+0000 7fb154758700 1 -- 172.21.15.3:0/4058343491 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb1380a11d0 msgr2=0x7fb1380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.506+0000 7fb154758700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb1380a11d0 0x7fb1380a3690 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb14c067800 tx=0x7fb13c009260 comp rx=0 tx=0).stop 2023-07-20T01:42:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.506+0000 7fb154758700 1 -- 172.21.15.3:0/4058343491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14c100d40 msgr2=0x7fb14c19fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.506+0000 7fb154758700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14c100d40 0x7fb14c19fd20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb14400b780 tx=0x7fb14400bab0 comp rx=0 tx=0).stop 2023-07-20T01:42:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.506+0000 7fb154758700 1 -- 172.21.15.3:0/4058343491 shutdown_connections 2023-07-20T01:42:39.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.506+0000 7fb154758700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14c100d40 0x7fb14c19fd20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:39.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.506+0000 7fb154758700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb1380a11d0 0x7fb1380a3690 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:39.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.506+0000 7fb154758700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb14c0ff9e0 0x7fb14c0ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:39.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.506+0000 7fb154758700 1 --2- 172.21.15.3:0/4058343491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb14c100380 0x7fb14c19d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:39.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.506+0000 7fb154758700 1 -- 172.21.15.3:0/4058343491 >> 172.21.15.3:0/4058343491 conn(0x7fb14c0fa450 msgr2=0x7fb14c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:39.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.506+0000 7fb154758700 1 -- 172.21.15.3:0/4058343491 shutdown_connections 2023-07-20T01:42:39.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.506+0000 7fb154758700 1 -- 172.21.15.3:0/4058343491 wait complete. 2023-07-20T01:42:39.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-20T01:42:39.501 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:39 smithi043 ceph-mon[97823]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T01:42:39.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:39 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/717047014' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T01:42:39.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:39 smithi043 ceph-mon[97823]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T01:42:39.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:39 smithi043 ceph-mon[97823]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:39.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:39 smithi043 ceph-mon[97823]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2023-07-20T01:42:39.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:39 smithi043 ceph-mon[97823]: daemon mds.b is now active in filesystem cephfs as rank 1 2023-07-20T01:42:39.502 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:39 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/4058343491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T01:42:39.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:39 smithi086 ceph-mon[98138]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T01:42:39.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:39 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/717047014' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T01:42:39.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:39 smithi086 ceph-mon[98138]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T01:42:39.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:39 smithi086 ceph-mon[98138]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T01:42:39.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:39 smithi086 ceph-mon[98138]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2023-07-20T01:42:39.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:39 smithi086 ceph-mon[98138]: daemon mds.b is now active in filesystem cephfs as rank 1 2023-07-20T01:42:39.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:39 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/4058343491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T01:42:39.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.659+0000 7fb8d05ea700 1 -- 172.21.15.3:0/877672341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c8100930 msgr2=0x7fb8c8100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:39.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.659+0000 7fb8d05ea700 1 --2- 172.21.15.3:0/877672341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c8100930 0x7fb8c8100d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40099c0 tx=0x7fb8c4009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:39.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.659+0000 7fb8d05ea700 1 -- 172.21.15.3:0/877672341 shutdown_connections 2023-07-20T01:42:39.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.659+0000 7fb8d05ea700 1 --2- 172.21.15.3:0/877672341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c81012d0 0x7fb8c8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:39.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.659+0000 7fb8d05ea700 1 --2- 172.21.15.3:0/877672341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c80ff740 0x7fb8c80ffb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:39.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.659+0000 7fb8d05ea700 1 --2- 172.21.15.3:0/877672341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c8100930 0x7fb8c8100d90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:39.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.659+0000 7fb8d05ea700 1 -- 172.21.15.3:0/877672341 >> 172.21.15.3:0/877672341 conn(0x7fb8c80facf0 msgr2=0x7fb8c80fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:39.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.660+0000 7fb8d05ea700 1 -- 172.21.15.3:0/877672341 shutdown_connections 2023-07-20T01:42:39.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.660+0000 7fb8d05ea700 1 -- 172.21.15.3:0/877672341 wait complete. 2023-07-20T01:42:39.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.660+0000 7fb8d05ea700 1 Processor -- start 2023-07-20T01:42:39.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.660+0000 7fb8d05ea700 1 -- start start 2023-07-20T01:42:39.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.660+0000 7fb8d05ea700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c80ff740 0x7fb8c8195440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:39.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.660+0000 7fb8d05ea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c8100930 0x7fb8c8195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:39.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.660+0000 7fb8d05ea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c81012d0 0x7fb8c819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:39.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.660+0000 7fb8d05ea700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c819cfc0 con 0x7fb8c8100930 2023-07-20T01:42:39.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.660+0000 7fb8d05ea700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c819d100 con 0x7fb8c80ff740 2023-07-20T01:42:39.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.660+0000 7fb8d05ea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c819d290 con 0x7fb8c81012d0 2023-07-20T01:42:39.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.660+0000 7fb8cdb85700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c8100930 0x7fb8c8195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:39.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8cdb85700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c8100930 0x7fb8c8195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42838/0 (socket says 172.21.15.3:42838) 2023-07-20T01:42:39.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8cdb85700 1 -- 172.21.15.3:0/3107238873 learned_addr learned my addr 172.21.15.3:0/3107238873 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:39.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8ceb87700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c81012d0 0x7fb8c819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:39.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8ce386700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c80ff740 0x7fb8c8195440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:39.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8cdb85700 1 -- 172.21.15.3:0/3107238873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c81012d0 msgr2=0x7fb8c819c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:39.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8cdb85700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c81012d0 0x7fb8c819c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:39.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8cdb85700 1 -- 172.21.15.3:0/3107238873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c80ff740 msgr2=0x7fb8c8195440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:39.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8cdb85700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c80ff740 0x7fb8c8195440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:39.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8cdb85700 1 -- 172.21.15.3:0/3107238873 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c40095f0 con 0x7fb8c8100930 2023-07-20T01:42:39.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8ce386700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c80ff740 0x7fb8c8195440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:42:39.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8cdb85700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c8100930 0x7fb8c8195980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009990 tx=0x7fb8c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:39.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8bb7fe700 1 -- 172.21.15.3:0/3107238873 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c401d070 con 0x7fb8c8100930 2023-07-20T01:42:39.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8bb7fe700 1 -- 172.21.15.3:0/3107238873 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8c4004090 con 0x7fb8c8100930 2023-07-20T01:42:39.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8d05ea700 1 -- 172.21.15.3:0/3107238873 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c819d510 con 0x7fb8c8100930 2023-07-20T01:42:39.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8bb7fe700 1 -- 172.21.15.3:0/3107238873 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4004a00 con 0x7fb8c8100930 2023-07-20T01:42:39.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.661+0000 7fb8d05ea700 1 -- 172.21.15.3:0/3107238873 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c819d9c0 con 0x7fb8c8100930 2023-07-20T01:42:39.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.662+0000 7fb8d05ea700 1 -- 172.21.15.3:0/3107238873 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c8106d00 con 0x7fb8c8100930 2023-07-20T01:42:39.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.663+0000 7fb8bb7fe700 1 -- 172.21.15.3:0/3107238873 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fb8c4003bd0 con 0x7fb8c8100930 2023-07-20T01:42:39.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.664+0000 7fb8bb7fe700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb8b40a1150 0x7fb8b40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:39.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.664+0000 7fb8ce386700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb8b40a1150 0x7fb8b40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:39.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.664+0000 7fb8bb7fe700 1 -- 172.21.15.3:0/3107238873 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb8c40cdb80 con 0x7fb8c8100930 2023-07-20T01:42:39.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.664+0000 7fb8ce386700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb8b40a1150 0x7fb8b40a3610 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc005950 tx=0x7fb8bc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:39.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.667+0000 7fb8bb7fe700 1 -- 172.21.15.3:0/3107238873 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb8c4097410 con 0x7fb8c8100930 2023-07-20T01:42:39.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:39.913+0000 7fb8d05ea700 1 -- 172.21.15.3:0/3107238873 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fb8c802c970 con 0x7fb8c8100930 2023-07-20T01:42:40.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.520+0000 7fb8bb7fe700 1 -- 172.21.15.3:0/3107238873 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7fb8c409aa30 con 0x7fb8c8100930 2023-07-20T01:42:40.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.522+0000 7fb8d05ea700 1 -- 172.21.15.3:0/3107238873 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb8b40a1150 msgr2=0x7fb8b40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:40.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.522+0000 7fb8d05ea700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb8b40a1150 0x7fb8b40a3610 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc005950 tx=0x7fb8bc0058e0 comp rx=0 tx=0).stop 2023-07-20T01:42:40.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.522+0000 7fb8d05ea700 1 -- 172.21.15.3:0/3107238873 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c8100930 msgr2=0x7fb8c8195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:40.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.522+0000 7fb8d05ea700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c8100930 0x7fb8c8195980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009990 tx=0x7fb8c4004730 comp rx=0 tx=0).stop 2023-07-20T01:42:40.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.523+0000 7fb8d05ea700 1 -- 172.21.15.3:0/3107238873 shutdown_connections 2023-07-20T01:42:40.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.523+0000 7fb8d05ea700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c81012d0 0x7fb8c819c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:40.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.523+0000 7fb8d05ea700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7fb8b40a1150 0x7fb8b40a3610 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:40.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.523+0000 7fb8d05ea700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8c80ff740 0x7fb8c8195440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:40.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.523+0000 7fb8d05ea700 1 --2- 172.21.15.3:0/3107238873 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c8100930 0x7fb8c8195980 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:40.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.523+0000 7fb8d05ea700 1 -- 172.21.15.3:0/3107238873 >> 172.21.15.3:0/3107238873 conn(0x7fb8c80facf0 msgr2=0x7fb8c8103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:40.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.523+0000 7fb8d05ea700 1 -- 172.21.15.3:0/3107238873 shutdown_connections 2023-07-20T01:42:40.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.523+0000 7fb8d05ea700 1 -- 172.21.15.3:0/3107238873 wait complete. 2023-07-20T01:42:40.509 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2023-07-20T01:42:40.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2023-07-20T01:42:40.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.678+0000 7f95f90d2700 1 -- 172.21.15.3:0/582431527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f4100930 msgr2=0x7f95f4100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:40.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.678+0000 7f95f90d2700 1 --2- 172.21.15.3:0/582431527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f4100930 0x7f95f4100d90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f95e40099c0 tx=0x7f95e4009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:40.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.678+0000 7f95f90d2700 1 -- 172.21.15.3:0/582431527 shutdown_connections 2023-07-20T01:42:40.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.678+0000 7f95f90d2700 1 --2- 172.21.15.3:0/582431527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f41012d0 0x7f95f4107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:40.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.678+0000 7f95f90d2700 1 --2- 172.21.15.3:0/582431527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95f40ff740 0x7f95f40ffb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:40.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.678+0000 7f95f90d2700 1 --2- 172.21.15.3:0/582431527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f4100930 0x7f95f4100d90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:40.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.678+0000 7f95f90d2700 1 -- 172.21.15.3:0/582431527 >> 172.21.15.3:0/582431527 conn(0x7f95f40facf0 msgr2=0x7f95f40fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:40.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.678+0000 7f95f90d2700 1 -- 172.21.15.3:0/582431527 shutdown_connections 2023-07-20T01:42:40.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.678+0000 7f95f90d2700 1 -- 172.21.15.3:0/582431527 wait complete. 2023-07-20T01:42:40.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f90d2700 1 Processor -- start 2023-07-20T01:42:40.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f90d2700 1 -- start start 2023-07-20T01:42:40.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f90d2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95f40ff740 0x7f95f41965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f90d2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f4100930 0x7f95f4196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f90d2700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f41012d0 0x7f95f419db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f90d2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f419e140 con 0x7f95f4100930 2023-07-20T01:42:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f90d2700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f419e280 con 0x7f95f41012d0 2023-07-20T01:42:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f90d2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f419e410 con 0x7f95f40ff740 2023-07-20T01:42:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f4100930 0x7f95f4196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f4100930 0x7f95f4196b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42852/0 (socket says 172.21.15.3:42852) 2023-07-20T01:42:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f259c700 1 -- 172.21.15.3:0/2226738001 learned_addr learned my addr 172.21.15.3:0/2226738001 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f2d9d700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95f40ff740 0x7f95f41965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.679+0000 7f95f359e700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f41012d0 0x7f95f419db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.680+0000 7f95f259c700 1 -- 172.21.15.3:0/2226738001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95f40ff740 msgr2=0x7f95f41965c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.680+0000 7f95f259c700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95f40ff740 0x7f95f41965c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.680+0000 7f95f259c700 1 -- 172.21.15.3:0/2226738001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f41012d0 msgr2=0x7f95f419db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.680+0000 7f95f259c700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f41012d0 0x7f95f419db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:40.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.680+0000 7f95f259c700 1 -- 172.21.15.3:0/2226738001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e40095f0 con 0x7f95f4100930 2023-07-20T01:42:40.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.680+0000 7f95f259c700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f4100930 0x7f95f4196b00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f95e400c010 tx=0x7f95e4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:40.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.680+0000 7f95dbfff700 1 -- 172.21.15.3:0/2226738001 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e401d070 con 0x7f95f4100930 2023-07-20T01:42:40.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.680+0000 7f95dbfff700 1 -- 172.21.15.3:0/2226738001 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95e4004110 con 0x7f95f4100930 2023-07-20T01:42:40.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.680+0000 7f95f90d2700 1 -- 172.21.15.3:0/2226738001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f419e690 con 0x7f95f4100930 2023-07-20T01:42:40.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.680+0000 7f95dbfff700 1 -- 172.21.15.3:0/2226738001 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e4005360 con 0x7f95f4100930 2023-07-20T01:42:40.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.680+0000 7f95f90d2700 1 -- 172.21.15.3:0/2226738001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f419eab0 con 0x7f95f4100930 2023-07-20T01:42:40.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.681+0000 7f95f90d2700 1 -- 172.21.15.3:0/2226738001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f410b040 con 0x7f95f4100930 2023-07-20T01:42:40.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.682+0000 7f95dbfff700 1 -- 172.21.15.3:0/2226738001 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f95e4004a10 con 0x7f95f4100930 2023-07-20T01:42:40.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.683+0000 7f95dbfff700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f95dc0a11d0 0x7f95dc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:40.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.683+0000 7f95f2d9d700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f95dc0a11d0 0x7f95dc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:40.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.683+0000 7f95dbfff700 1 -- 172.21.15.3:0/2226738001 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f95e40cd690 con 0x7f95f4100930 2023-07-20T01:42:40.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.683+0000 7f95f2d9d700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f95dc0a11d0 0x7f95dc0a3690 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f95e8005950 tx=0x7f95e80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:40.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.687+0000 7f95dbfff700 1 -- 172.21.15.3:0/2226738001 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f95e4096f20 con 0x7f95f4100930 2023-07-20T01:42:40.751 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:40 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.43:6838/3251120203,v1:172.21.15.43:6839/3251120203] up:active 2023-07-20T01:42:40.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:40 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/4058343491' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T01:42:40.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:40 smithi043 ceph-mon[97823]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T01:42:40.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:40 smithi043 ceph-mon[97823]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2023-07-20T01:42:40.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:40 smithi043 ceph-mon[97823]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2023-07-20T01:42:40.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:40 smithi043 ceph-mon[97823]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-07-20T01:42:40.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:40 smithi043 ceph-mon[97823]: pgmap v252: 97 pgs: 97 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 10 op/s 2023-07-20T01:42:40.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:40 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3107238873' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T01:42:40.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:40 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.43:6838/3251120203,v1:172.21.15.43:6839/3251120203] up:active 2023-07-20T01:42:40.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:40 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/4058343491' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T01:42:40.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:40 smithi086 ceph-mon[98138]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T01:42:40.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:40 smithi086 ceph-mon[98138]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2023-07-20T01:42:40.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:40 smithi086 ceph-mon[98138]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2023-07-20T01:42:40.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:40 smithi086 ceph-mon[98138]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-07-20T01:42:40.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:40 smithi086 ceph-mon[98138]: pgmap v252: 97 pgs: 97 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 10 op/s 2023-07-20T01:42:40.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:40 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3107238873' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T01:42:40.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:40.918+0000 7f95f90d2700 1 -- 172.21.15.3:0/2226738001 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f95f40fbc80 con 0x7f95dc0a11d0 2023-07-20T01:42:40.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.43:6838/3251120203,v1:172.21.15.43:6839/3251120203] up:active 2023-07-20T01:42:40.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/4058343491' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T01:42:40.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-mon[96422]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T01:42:40.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-mon[96422]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2023-07-20T01:42:40.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-mon[96422]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2023-07-20T01:42:40.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-mon[96422]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-07-20T01:42:40.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-mon[96422]: pgmap v252: 97 pgs: 97 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 10 op/s 2023-07-20T01:42:40.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3107238873' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T01:42:40.894 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:42:40.923+0000 7fbd6b83c700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T01:42:40.894 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:42:40.923+0000 7fbd6b83c700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T01:42:40.894 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:42:40.923+0000 7fbd6b83c700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T01:42:40.894 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:42:40.923+0000 7fbd6b83c700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T01:42:40.894 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:42:40 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:42:40.923+0000 7fbd6b83c700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T01:42:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.553+0000 7f95dbfff700 1 -- 172.21.15.3:0/2226738001 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f95f40fbc80 con 0x7f95dc0a11d0 2023-07-20T01:42:41.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.555+0000 7f95f90d2700 1 -- 172.21.15.3:0/2226738001 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f95dc0a11d0 msgr2=0x7f95dc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:41.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.555+0000 7f95f90d2700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f95dc0a11d0 0x7f95dc0a3690 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f95e8005950 tx=0x7f95e80058e0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.555+0000 7f95f90d2700 1 -- 172.21.15.3:0/2226738001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f4100930 msgr2=0x7f95f4196b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.555+0000 7f95f90d2700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f4100930 0x7f95f4196b00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f95e400c010 tx=0x7f95e4009580 comp rx=0 tx=0).stop 2023-07-20T01:42:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.555+0000 7f95f90d2700 1 -- 172.21.15.3:0/2226738001 shutdown_connections 2023-07-20T01:42:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.555+0000 7f95f90d2700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f95dc0a11d0 0x7f95dc0a3690 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.555+0000 7f95f90d2700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f41012d0 0x7f95f419db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.555+0000 7f95f90d2700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95f40ff740 0x7f95f41965c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.555+0000 7f95f90d2700 1 --2- 172.21.15.3:0/2226738001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f4100930 0x7f95f4196b00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.555+0000 7f95f90d2700 1 -- 172.21.15.3:0/2226738001 >> 172.21.15.3:0/2226738001 conn(0x7f95f40facf0 msgr2=0x7f95f40fb570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.556+0000 7f95f90d2700 1 -- 172.21.15.3:0/2226738001 shutdown_connections 2023-07-20T01:42:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.556+0000 7f95f90d2700 1 -- 172.21.15.3:0/2226738001 wait complete. 2023-07-20T01:42:41.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs --group_name=qa sv_0 '' 2023-07-20T01:42:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.708+0000 7f03ff71a700 1 -- 172.21.15.3:0/3222328080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f80ff740 msgr2=0x7f03f80ffb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.708+0000 7f03ff71a700 1 --2- 172.21.15.3:0/3222328080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f80ff740 0x7f03f80ffb60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f03e80099c0 tx=0x7f03e8009cd0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.709+0000 7f03ff71a700 1 -- 172.21.15.3:0/3222328080 shutdown_connections 2023-07-20T01:42:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.709+0000 7f03ff71a700 1 --2- 172.21.15.3:0/3222328080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03f81012d0 0x7f03f8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.709+0000 7f03ff71a700 1 --2- 172.21.15.3:0/3222328080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f80ff740 0x7f03f80ffb60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.709+0000 7f03ff71a700 1 --2- 172.21.15.3:0/3222328080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03f8100930 0x7f03f8100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.709+0000 7f03ff71a700 1 -- 172.21.15.3:0/3222328080 >> 172.21.15.3:0/3222328080 conn(0x7f03f80facf0 msgr2=0x7f03f80fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.709+0000 7f03ff71a700 1 -- 172.21.15.3:0/3222328080 shutdown_connections 2023-07-20T01:42:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.709+0000 7f03ff71a700 1 -- 172.21.15.3:0/3222328080 wait complete. 2023-07-20T01:42:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.709+0000 7f03ff71a700 1 Processor -- start 2023-07-20T01:42:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.709+0000 7f03ff71a700 1 -- start start 2023-07-20T01:42:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03ff71a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f80ff740 0x7f03f810a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03ff71a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03f8100930 0x7f03f8108690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03ff71a700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03f81012d0 0x7f03f8108bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03ff71a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8109370 con 0x7f03f80ff740 2023-07-20T01:42:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03ff71a700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f81094e0 con 0x7f03f81012d0 2023-07-20T01:42:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03ff71a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f81a0620 con 0x7f03f8100930 2023-07-20T01:42:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03fd4b6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f80ff740 0x7f03f810a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03fd4b6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f80ff740 0x7f03f810a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42882/0 (socket says 172.21.15.3:42882) 2023-07-20T01:42:41.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03fd4b6700 1 -- 172.21.15.3:0/3902548056 learned_addr learned my addr 172.21.15.3:0/3902548056 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:41.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03fccb5700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03f8100930 0x7f03f8108690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03fdcb7700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03f81012d0 0x7f03f8108bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03fd4b6700 1 -- 172.21.15.3:0/3902548056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03f8100930 msgr2=0x7f03f8108690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03fd4b6700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03f8100930 0x7f03f8108690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03fd4b6700 1 -- 172.21.15.3:0/3902548056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03f81012d0 msgr2=0x7f03f8108bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03fd4b6700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03f81012d0 0x7f03f8108bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03fd4b6700 1 -- 172.21.15.3:0/3902548056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e80095f0 con 0x7f03f80ff740 2023-07-20T01:42:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03fd4b6700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f80ff740 0x7f03f810a050 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009f50 tx=0x7f03e8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:41.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03ee7fc700 1 -- 172.21.15.3:0/3902548056 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e801d070 con 0x7f03f80ff740 2023-07-20T01:42:41.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03ff71a700 1 -- 172.21.15.3:0/3902548056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f81a08a0 con 0x7f03f80ff740 2023-07-20T01:42:41.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03ee7fc700 1 -- 172.21.15.3:0/3902548056 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03e8004010 con 0x7f03f80ff740 2023-07-20T01:42:41.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03ee7fc700 1 -- 172.21.15.3:0/3902548056 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e800fdc0 con 0x7f03f80ff740 2023-07-20T01:42:41.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.710+0000 7f03ff71a700 1 -- 172.21.15.3:0/3902548056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f81a0cd0 con 0x7f03f80ff740 2023-07-20T01:42:41.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.711+0000 7f03ff71a700 1 -- 172.21.15.3:0/3902548056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f8106d30 con 0x7f03f80ff740 2023-07-20T01:42:41.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.712+0000 7f03ee7fc700 1 -- 172.21.15.3:0/3902548056 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f03e800f4f0 con 0x7f03f80ff740 2023-07-20T01:42:41.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.713+0000 7f03ee7fc700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f03e40a1150 0x7f03e40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:41.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.713+0000 7f03fccb5700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f03e40a1150 0x7f03e40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.714+0000 7f03ee7fc700 1 -- 172.21.15.3:0/3902548056 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f03e80cdb80 con 0x7f03f80ff740 2023-07-20T01:42:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.714+0000 7f03fccb5700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f03e40a1150 0x7f03e40a3610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f03f4005950 tx=0x7f03f400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.717+0000 7f03ee7fc700 1 -- 172.21.15.3:0/3902548056 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f03e8097410 con 0x7f03f80ff740 2023-07-20T01:42:41.751 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:41 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.86:6836/2620198024,v1:172.21.15.86:6837/2620198024] up:active 2023-07-20T01:42:41.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:41 smithi043 ceph-mon[97823]: from='client.? 172.21.15.3:0/3107238873' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T01:42:41.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:41 smithi043 ceph-mon[97823]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T01:42:41.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:41 smithi043 ceph-mon[97823]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2023-07-20T01:42:41.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:41 smithi043 ceph-mon[97823]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2023-07-20T01:42:41.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:41 smithi043 ceph-mon[97823]: daemon mds.h is now active in filesystem cephfs as rank 3 2023-07-20T01:42:41.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:41 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.86:6836/2620198024,v1:172.21.15.86:6837/2620198024] up:active 2023-07-20T01:42:41.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:41 smithi086 ceph-mon[98138]: from='client.? 172.21.15.3:0/3107238873' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T01:42:41.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:41 smithi086 ceph-mon[98138]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T01:42:41.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:41 smithi086 ceph-mon[98138]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2023-07-20T01:42:41.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:41 smithi086 ceph-mon[98138]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2023-07-20T01:42:41.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:41 smithi086 ceph-mon[98138]: daemon mds.h is now active in filesystem cephfs as rank 3 2023-07-20T01:42:41.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.948+0000 7f03ff71a700 1 -- 172.21.15.3:0/3902548056 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "group_name": "", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f03f8060bb0 con 0x7f03e40a1150 2023-07-20T01:42:41.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:41 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.86:6836/2620198024,v1:172.21.15.86:6837/2620198024] up:active 2023-07-20T01:42:41.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:41 smithi003 ceph-mon[96422]: from='client.? 172.21.15.3:0/3107238873' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T01:42:41.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:41 smithi003 ceph-mon[96422]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T01:42:41.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:41 smithi003 ceph-mon[96422]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2023-07-20T01:42:41.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:41 smithi003 ceph-mon[96422]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2023-07-20T01:42:41.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:41 smithi003 ceph-mon[96422]: daemon mds.h is now active in filesystem cephfs as rank 3 2023-07-20T01:42:41.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.977+0000 7f03ee7fc700 1 -- 172.21.15.3:0/3902548056 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f03f8060bb0 con 0x7f03e40a1150 2023-07-20T01:42:41.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.979+0000 7f03ff71a700 1 -- 172.21.15.3:0/3902548056 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f03e40a1150 msgr2=0x7f03e40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:41.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.979+0000 7f03ff71a700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f03e40a1150 0x7f03e40a3610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f03f4005950 tx=0x7f03f400b410 comp rx=0 tx=0).stop 2023-07-20T01:42:41.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.979+0000 7f03ff71a700 1 -- 172.21.15.3:0/3902548056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f80ff740 msgr2=0x7f03f810a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:41.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.979+0000 7f03ff71a700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f80ff740 0x7f03f810a050 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009f50 tx=0x7f03e8005740 comp rx=0 tx=0).stop 2023-07-20T01:42:41.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.979+0000 7f03ff71a700 1 -- 172.21.15.3:0/3902548056 shutdown_connections 2023-07-20T01:42:41.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.979+0000 7f03ff71a700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f03e40a1150 0x7f03e40a3610 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.979+0000 7f03ff71a700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03f81012d0 0x7f03f8108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.979+0000 7f03ff71a700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f80ff740 0x7f03f810a050 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.979+0000 7f03ff71a700 1 --2- 172.21.15.3:0/3902548056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03f8100930 0x7f03f8108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:41.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.979+0000 7f03ff71a700 1 -- 172.21.15.3:0/3902548056 >> 172.21.15.3:0/3902548056 conn(0x7f03f80facf0 msgr2=0x7f03f8103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:41.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.979+0000 7f03ff71a700 1 -- 172.21.15.3:0/3902548056 shutdown_connections 2023-07-20T01:42:41.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:41.980+0000 7f03ff71a700 1 -- 172.21.15.3:0/3902548056 wait complete. 2023-07-20T01:42:41.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0 2023-07-20T01:42:42.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.123+0000 7f758c527700 1 -- 172.21.15.3:0/2628909075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75841012f0 msgr2=0x7f7584107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:42.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.123+0000 7f758c527700 1 --2- 172.21.15.3:0/2628909075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75841012f0 0x7f7584107ba0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f757c00b0d0 tx=0x7f757c00b3e0 comp rx=0 tx=0).stop 2023-07-20T01:42:42.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.123+0000 7f758c527700 1 -- 172.21.15.3:0/2628909075 shutdown_connections 2023-07-20T01:42:42.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.123+0000 7f758c527700 1 --2- 172.21.15.3:0/2628909075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75841012f0 0x7f7584107ba0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:42.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.123+0000 7f758c527700 1 --2- 172.21.15.3:0/2628909075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75840ff760 0x7f75840ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:42.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.123+0000 7f758c527700 1 --2- 172.21.15.3:0/2628909075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7584100950 0x7f7584100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:42.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.123+0000 7f758c527700 1 -- 172.21.15.3:0/2628909075 >> 172.21.15.3:0/2628909075 conn(0x7f75840facd0 msgr2=0x7f75840fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:42:42.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.123+0000 7f758c527700 1 -- 172.21.15.3:0/2628909075 shutdown_connections 2023-07-20T01:42:42.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.123+0000 7f758c527700 1 -- 172.21.15.3:0/2628909075 wait complete. 2023-07-20T01:42:42.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758c527700 1 Processor -- start 2023-07-20T01:42:42.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758c527700 1 -- start start 2023-07-20T01:42:42.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758c527700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75840ff760 0x7f7584198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:42.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758c527700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7584100950 0x7f7584198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:42.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758c527700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75841012f0 0x7f758419fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:42.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758c527700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75841a0300 con 0x7f75840ff760 2023-07-20T01:42:42.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758c527700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75841a0440 con 0x7f75841012f0 2023-07-20T01:42:42.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758c527700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75841a05d0 con 0x7f7584100950 2023-07-20T01:42:42.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758a2c3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75840ff760 0x7f7584198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:42.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758a2c3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75840ff760 0x7f7584198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42898/0 (socket says 172.21.15.3:42898) 2023-07-20T01:42:42.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758a2c3700 1 -- 172.21.15.3:0/2942065072 learned_addr learned my addr 172.21.15.3:0/2942065072 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-07-20T01:42:42.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f7589ac2700 1 --2- 172.21.15.3:0/2942065072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7584100950 0x7f7584198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:42.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758aac4700 1 --2- 172.21.15.3:0/2942065072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75841012f0 0x7f758419fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:42.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758a2c3700 1 -- 172.21.15.3:0/2942065072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7584100950 msgr2=0x7f7584198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:42.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758a2c3700 1 --2- 172.21.15.3:0/2942065072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7584100950 0x7f7584198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:42.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.124+0000 7f758a2c3700 1 -- 172.21.15.3:0/2942065072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75841012f0 msgr2=0x7f758419fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:42:42.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.125+0000 7f758a2c3700 1 --2- 172.21.15.3:0/2942065072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75841012f0 0x7f758419fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:42:42.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.125+0000 7f758a2c3700 1 -- 172.21.15.3:0/2942065072 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c009d00 con 0x7f75840ff760 2023-07-20T01:42:42.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.125+0000 7f758aac4700 1 --2- 172.21.15.3:0/2942065072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75841012f0 0x7f758419fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:42:42.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.125+0000 7f758a2c3700 1 --2- 172.21.15.3:0/2942065072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75840ff760 0x7f7584198780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f758000c990 tx=0x7f758000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:42.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.125+0000 7f75777fe700 1 -- 172.21.15.3:0/2942065072 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580007b00 con 0x7f75840ff760 2023-07-20T01:42:42.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.125+0000 7f758c527700 1 -- 172.21.15.3:0/2942065072 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75841a08b0 con 0x7f75840ff760 2023-07-20T01:42:42.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.125+0000 7f75777fe700 1 -- 172.21.15.3:0/2942065072 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7580007c60 con 0x7f75840ff760 2023-07-20T01:42:42.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.125+0000 7f75777fe700 1 -- 172.21.15.3:0/2942065072 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758000fea0 con 0x7f75840ff760 2023-07-20T01:42:42.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.125+0000 7f758c527700 1 -- 172.21.15.3:0/2942065072 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75841a0e10 con 0x7f75840ff760 2023-07-20T01:42:42.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.126+0000 7f758c527700 1 -- 172.21.15.3:0/2942065072 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758410afd0 con 0x7f75840ff760 2023-07-20T01:42:42.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.128+0000 7f75777fe700 1 -- 172.21.15.3:0/2942065072 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f758000f450 con 0x7f75840ff760 2023-07-20T01:42:42.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.128+0000 7f75777fe700 1 --2- 172.21.15.3:0/2942065072 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f75700a1150 0x7f75700a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:42:42.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.128+0000 7f7589ac2700 1 --2- 172.21.15.3:0/2942065072 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f75700a1150 0x7f75700a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:42:42.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.128+0000 7f75777fe700 1 -- 172.21.15.3:0/2942065072 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f75800cc880 con 0x7f75840ff760 2023-07-20T01:42:42.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.129+0000 7f7589ac2700 1 --2- 172.21.15.3:0/2942065072 >> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] conn(0x7f75700a1150 0x7f75700a3610 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7578005950 tx=0x7f757800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:42:42.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.132+0000 7f75777fe700 1 -- 172.21.15.3:0/2942065072 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f75800961b0 con 0x7f75840ff760 2023-07-20T01:42:42.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.368+0000 7f758c527700 1 -- 172.21.15.3:0/2942065072 --> [v2:172.21.15.3:6800/4201887131,v1:172.21.15.3:6801/4201887131] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f7584068ef0 con 0x7f75700a1150 2023-07-20T01:42:42.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-07-20T01:42:42.370+0000 7f75777fe700 1 -- 172.21.15.3:0/2942065072 <== mgr.14152 v2:172.21.15.3:6800/4201887131 1 ==== mgr_command_reply(tid 0: -2 subvolume 'sv_0' does not exist) v1 ==== 39+0+0 (secure 0 0 0) 0x7f7584068ef0 con 0x7f75700a1150 2023-07-20T01:42:42.339 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENOENT: subvolume 'sv_0' does not exist 2023-07-20T01:42:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-20T01:42:42.343 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T01:42:42.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:42:42.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:42:42.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:42:42.452 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-20T01:42:42.453 DEBUG:teuthology.orchestra.run.smithi003:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T01:42:42.473 DEBUG:teuthology.orchestra.run.smithi043:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T01:42:42.493 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T01:42:42.516 INFO:tasks.cephadm:Stopping all daemons... 2023-07-20T01:42:42.517 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-20T01:42:42.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.a 2023-07-20T01:42:42.686 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:42:42.370+0000 7fbd6b83c700 -1 mgr.server reply reply (2) No such file or directory subvolume 'sv_0' does not exist 2023-07-20T01:42:42.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-mon[96422]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:42:42.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.43:6836/1379813430,v1:172.21.15.43:6837/1379813430] up:active 2023-07-20T01:42:42.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-mon[96422]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T01:42:42.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-mon[96422]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T01:42:42.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-mon[96422]: Cluster is now healthy 2023-07-20T01:42:42.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-mon[96422]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2023-07-20T01:42:42.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-mon[96422]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-20T01:42:42.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-mon[96422]: daemon mds.f is now active in filesystem cephfs as rank 4 2023-07-20T01:42:42.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-mon[96422]: pgmap v253: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s wr, 13 op/s 2023-07-20T01:42:42.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-mon[96422]: mds.? [v2:172.21.15.86:6834/113908943,v1:172.21.15.86:6835/113908943] up:active 2023-07-20T01:42:42.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-mon[96422]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-20T01:42:42.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 systemd[1]: Stopping Ceph mon.a for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:42.751 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:42 smithi043 ceph-mon[97823]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:42:42.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:42 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.43:6836/1379813430,v1:172.21.15.43:6837/1379813430] up:active 2023-07-20T01:42:42.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:42 smithi043 ceph-mon[97823]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T01:42:42.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:42 smithi043 ceph-mon[97823]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T01:42:42.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:42 smithi043 ceph-mon[97823]: Cluster is now healthy 2023-07-20T01:42:42.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:42 smithi043 ceph-mon[97823]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2023-07-20T01:42:42.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:42 smithi043 ceph-mon[97823]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-20T01:42:42.752 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:42 smithi043 ceph-mon[97823]: daemon mds.f is now active in filesystem cephfs as rank 4 2023-07-20T01:42:42.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:42 smithi043 ceph-mon[97823]: pgmap v253: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s wr, 13 op/s 2023-07-20T01:42:42.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:42 smithi043 ceph-mon[97823]: mds.? [v2:172.21.15.86:6834/113908943,v1:172.21.15.86:6835/113908943] up:active 2023-07-20T01:42:42.753 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:42 smithi043 ceph-mon[97823]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-20T01:42:42.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-mon[98138]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:42:42.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.43:6836/1379813430,v1:172.21.15.43:6837/1379813430] up:active 2023-07-20T01:42:42.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-mon[98138]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T01:42:42.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-mon[98138]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T01:42:42.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-mon[98138]: Cluster is now healthy 2023-07-20T01:42:42.852 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-mon[98138]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2023-07-20T01:42:42.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-mon[98138]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-20T01:42:42.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-mon[98138]: daemon mds.f is now active in filesystem cephfs as rank 4 2023-07-20T01:42:42.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-mon[98138]: pgmap v253: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s wr, 13 op/s 2023-07-20T01:42:42.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-mon[98138]: mds.? [v2:172.21.15.86:6834/113908943,v1:172.21.15.86:6835/113908943] up:active 2023-07-20T01:42:42.853 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-mon[98138]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-20T01:42:43.172 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-a[96398]: 2023-07-20T01:42:42.906+0000 7f6002021700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:42:43.173 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:42 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-a[96398]: 2023-07-20T01:42:42.906+0000 7f6002021700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-20T01:42:43.351 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-i[124445]: 2023-07-20T01:42:42.949+0000 7f1001dc7700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-07-20T01:42:43.352 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:42:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-i[124445]: 2023-07-20T01:42:42.949+0000 7f1001dc7700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-07-20T01:42:43.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.a.service' 2023-07-20T01:42:43.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:43 smithi003 bash[133142]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-a 2023-07-20T01:42:43.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:43 smithi003 systemd[1]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.a.service: Succeeded. 2023-07-20T01:42:43.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Jul 20 01:42:43 smithi003 systemd[1]: Stopped Ceph mon.a for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:43.479 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:42:43.480 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-20T01:42:43.481 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-20T01:42:43.482 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.b 2023-07-20T01:42:43.842 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:43 smithi043 systemd[1]: Stopping Ceph mon.b for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:44.251 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-b[97799]: 2023-07-20T01:42:43.873+0000 7f8c8af6e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:42:44.252 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-b[97799]: 2023-07-20T01:42:43.874+0000 7f8c8af6e700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-20T01:42:44.751 INFO:journalctl@ceph.mon.b.smithi043.stdout:Jul 20 01:42:44 smithi043 bash[124312]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-b 2023-07-20T01:42:44.968 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.b.service' 2023-07-20T01:42:45.022 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:42:45.022 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-20T01:42:45.023 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-20T01:42:45.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.c 2023-07-20T01:42:45.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:45 smithi086 systemd[1]: Stopping Ceph mon.c for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:45.659 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:45 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-c[98110]: 2023-07-20T01:42:45.419+0000 7f2a42801700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:42:45.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:45 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-c[98110]: 2023-07-20T01:42:45.419+0000 7f2a42801700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-20T01:42:45.943 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jul 20 01:42:45 smithi086 bash[125851]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mon-c 2023-07-20T01:42:45.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mon.c.service' 2023-07-20T01:42:45.999 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:42:45.999 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-20T01:42:46.000 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-20T01:42:46.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T01:42:46.340 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:42:46 smithi003 systemd[1]: Stopping Ceph mgr.x for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:46.622 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:42:46 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x[96672]: 2023-07-20T01:42:46.372+0000 7fbdfd470700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:42:46.895 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Jul 20 01:42:46 smithi003 bash[133656]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-x 2023-07-20T01:42:46.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.x.service' 2023-07-20T01:42:46.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:42:46.944 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-20T01:42:46.945 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-20T01:42:46.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.y 2023-07-20T01:42:47.251 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:42:47 smithi043 systemd[1]: Stopping Ceph mgr.y for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:47.667 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:42:47 smithi043 bash[124694]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-y 2023-07-20T01:42:47.854 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Jul 20 01:42:47 smithi043 systemd[1]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-20T01:42:47.859 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.y.service' 2023-07-20T01:42:47.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:42:47.901 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-20T01:42:47.902 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-20T01:42:47.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.z 2023-07-20T01:42:48.316 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:42:47 smithi086 systemd[1]: Stopping Ceph mgr.z for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:48.601 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:42:48 smithi086 bash[126027]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mgr-z 2023-07-20T01:42:48.602 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Jul 20 01:42:48 smithi086 systemd[1]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-20T01:42:48.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mgr.z.service' 2023-07-20T01:42:48.733 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:42:48.734 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-20T01:42:48.734 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-20T01:42:48.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.0 2023-07-20T01:42:49.098 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:42:48 smithi003 systemd[1]: Stopping Ceph osd.0 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:49.436 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:42:49 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-0[109510]: 2023-07-20T01:42:49.130+0000 7ff99a07e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:42:49.436 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:42:49 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-0[109510]: 2023-07-20T01:42:49.130+0000 7ff99a07e700 -1 osd.0 80 *** Got signal Terminated *** 2023-07-20T01:42:49.436 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:42:49 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-0[109510]: 2023-07-20T01:42:49.130+0000 7ff99a07e700 -1 osd.0 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:42:54.775 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:42:54 smithi003 bash[133796]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-0 2023-07-20T01:42:55.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.0.service' 2023-07-20T01:42:55.974 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:42:55 smithi003 systemd[1]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.0.service: Succeeded. 2023-07-20T01:42:55.975 INFO:journalctl@ceph.osd.0.smithi003.stdout:Jul 20 01:42:55 smithi003 systemd[1]: Stopped Ceph osd.0 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:42:55.990 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:42:55.991 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-20T01:42:55.991 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-20T01:42:55.992 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.1 2023-07-20T01:42:56.355 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:42:56 smithi043 systemd[1]: Stopping Ceph osd.1 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:42:56.751 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:42:56 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-1[102965]: 2023-07-20T01:42:56.387+0000 7f879f212700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:42:56.752 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:42:56 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-1[102965]: 2023-07-20T01:42:56.387+0000 7f879f212700 -1 osd.1 80 *** Got signal Terminated *** 2023-07-20T01:42:56.752 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:42:56 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-1[102965]: 2023-07-20T01:42:56.387+0000 7f879f212700 -1 osd.1 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:43:02.251 INFO:journalctl@ceph.osd.1.smithi043.stdout:Jul 20 01:43:01 smithi043 bash[124844]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-1 2023-07-20T01:43:03.463 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.1.service' 2023-07-20T01:43:03.506 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:43:03.507 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-20T01:43:03.508 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-20T01:43:03.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.2 2023-07-20T01:43:03.851 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:43:03 smithi086 systemd[1]: Stopping Ceph osd.2 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:43:04.352 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:43:03 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-2[104035]: 2023-07-20T01:43:03.892+0000 7fce71686700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:43:04.352 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:43:03 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-2[104035]: 2023-07-20T01:43:03.892+0000 7fce71686700 -1 osd.2 80 *** Got signal Terminated *** 2023-07-20T01:43:04.353 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:43:03 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-2[104035]: 2023-07-20T01:43:03.892+0000 7fce71686700 -1 osd.2 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:43:09.851 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jul 20 01:43:09 smithi086 bash[126184]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-2 2023-07-20T01:43:10.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.2.service' 2023-07-20T01:43:10.855 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:43:10.856 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-20T01:43:10.857 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-20T01:43:10.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.3 2023-07-20T01:43:11.185 INFO:journalctl@ceph.osd.3.smithi003.stdout:Jul 20 01:43:10 smithi003 systemd[1]: Stopping Ceph osd.3 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:43:11.685 INFO:journalctl@ceph.osd.3.smithi003.stdout:Jul 20 01:43:11 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-3[114976]: 2023-07-20T01:43:11.243+0000 7f56327f6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:43:11.685 INFO:journalctl@ceph.osd.3.smithi003.stdout:Jul 20 01:43:11 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-3[114976]: 2023-07-20T01:43:11.243+0000 7f56327f6700 -1 osd.3 80 *** Got signal Terminated *** 2023-07-20T01:43:11.686 INFO:journalctl@ceph.osd.3.smithi003.stdout:Jul 20 01:43:11 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-3[114976]: 2023-07-20T01:43:11.243+0000 7f56327f6700 -1 osd.3 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:43:15.351 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:14 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:14.943+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:16.101 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:15 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:15.754+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:16.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:15 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:15.903+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:16.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:15 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:15.899+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:16.935 INFO:journalctl@ceph.osd.3.smithi003.stdout:Jul 20 01:43:16 smithi003 bash[134076]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-3 2023-07-20T01:43:17.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:16 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:16.874+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:17.101 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:16 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:16.778+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:17.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:16 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:16.853+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:18.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.3.service' 2023-07-20T01:43:18.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:17 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:17.902+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:18.102 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:17 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:17.808+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:18.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:43:18.122 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-20T01:43:18.123 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-20T01:43:18.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.4 2023-07-20T01:43:18.156 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:17 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:17.895+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:18.501 INFO:journalctl@ceph.osd.4.smithi043.stdout:Jul 20 01:43:18 smithi043 systemd[1]: Stopping Ceph osd.4 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:43:18.871 INFO:journalctl@ceph.osd.4.smithi043.stdout:Jul 20 01:43:18 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-4[108497]: 2023-07-20T01:43:18.545+0000 7f9bf87ee700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:43:18.871 INFO:journalctl@ceph.osd.4.smithi043.stdout:Jul 20 01:43:18 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-4[108497]: 2023-07-20T01:43:18.545+0000 7f9bf87ee700 -1 osd.4 80 *** Got signal Terminated *** 2023-07-20T01:43:18.871 INFO:journalctl@ceph.osd.4.smithi043.stdout:Jul 20 01:43:18 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-4[108497]: 2023-07-20T01:43:18.545+0000 7f9bf87ee700 -1 osd.4 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:43:18.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:18 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:18.677+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:19.101 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:18 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:18.778+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:19.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:18 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:18.851+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:19.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:18 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:18.853+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:19.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:18 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:18.905+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:20.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:19 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:19.830+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:20.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:19 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:19.827+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:20.103 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:19 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:19.782+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:20.185 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:19 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:19.727+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:20.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:19 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:19.878+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:20.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:20 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:20.700+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:20.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:20 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:20.687+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:21.101 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:20 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:20.734+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:21.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:20 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:20.869+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:21.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:20 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:20.811+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:21.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:20 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:20.924+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:21.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:21 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:21.700+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:21.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:21 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:21.653+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:22.101 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:21 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:21.725+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:22.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:21 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:21.895+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:22.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:21 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:21.788+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:22.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:21 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:21.967+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:22.905 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:22 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:22.580+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:22.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:22 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:22.696+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:22.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:22 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:22.653+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:23.101 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:22 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:22.678+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:23.101 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:22 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:22.678+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.282244+0000 front 2023-07-20T01:43:00.282091+0000 (oldest deadline 2023-07-20T01:43:22.582158+0000) 2023-07-20T01:43:23.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:22 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:22.905+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:23.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:22 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:22.781+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:23.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:22 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:22.781+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:23.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:22 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:22.939+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:23.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:22 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:22.939+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:23.932 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:23 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:23.611+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:23.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:23 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:23.686+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:23.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:23 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:23.703+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:24.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:23 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:23.856+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:24.102 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:23 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:23.716+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:24.102 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:23 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:23.716+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.282244+0000 front 2023-07-20T01:43:00.282091+0000 (oldest deadline 2023-07-20T01:43:22.582158+0000) 2023-07-20T01:43:24.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:23 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:23.748+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:24.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:23 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:23.748+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:24.192 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:23 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:23.965+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:24.192 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:23 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:23.966+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:24.501 INFO:journalctl@ceph.osd.4.smithi043.stdout:Jul 20 01:43:24 smithi043 bash[125131]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-4 2023-07-20T01:43:24.932 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:24 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:24.651+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:25.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:24 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:24.839+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:25.102 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:24 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:24.730+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:25.102 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:24 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:24.730+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.282244+0000 front 2023-07-20T01:43:00.282091+0000 (oldest deadline 2023-07-20T01:43:22.582158+0000) 2023-07-20T01:43:25.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:24 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:24.761+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:25.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:24 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:24.761+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:25.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:24 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:24.746+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:25.186 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:24 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:24.720+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:25.186 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:24 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:24.720+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:25.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:25 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:25.002+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:25.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:25 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:25.002+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:25.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.4.service' 2023-07-20T01:43:25.723 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:43:25.724 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-20T01:43:25.725 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-20T01:43:25.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.5 2023-07-20T01:43:25.930 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:25 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:25.687+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:25.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:25 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:25.715+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:25.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:25 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:25.732+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:25.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:25 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:25.732+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:26.064 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:25 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:25.783+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:26.065 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:25 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:25.783+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:26.065 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:25 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:25.814+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:26.065 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:25 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:25.814+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:26.065 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:25 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:25.690+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:26.066 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:25 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:25.690+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.282244+0000 front 2023-07-20T01:43:00.282091+0000 (oldest deadline 2023-07-20T01:43:22.582158+0000) 2023-07-20T01:43:26.066 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:25 smithi086 systemd[1]: Stopping Ceph osd.5 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:43:26.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:25 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:25.964+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:26.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:25 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:25.964+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:26.351 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:26 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:26.097+0000 7f8a72d5e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:43:26.351 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:26 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:26.097+0000 7f8a72d5e700 -1 osd.5 80 *** Got signal Terminated *** 2023-07-20T01:43:26.351 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:26 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:26.097+0000 7f8a72d5e700 -1 osd.5 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:43:26.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:26 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:26.758+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:26.936 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:26 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:26.758+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:26.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:26 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:26.694+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:26.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:26 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:26.694+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:27.001 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:26 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:26.711+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:27.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:26 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:26.947+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:27.002 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:26 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:26.947+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:27.101 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:26 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:26.665+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:27.102 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:26 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:26.665+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.282244+0000 front 2023-07-20T01:43:00.282091+0000 (oldest deadline 2023-07-20T01:43:22.582158+0000) 2023-07-20T01:43:27.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:26 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:26.830+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:27.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:26 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:26.830+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:27.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:26 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:26.792+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:27.103 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:26 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:26.792+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:27.951 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:27 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:27.721+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:28.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:27 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:27.835+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:28.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:27 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:27.835+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:28.102 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:27 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:27.716+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:28.102 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:27 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:27.716+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.282244+0000 front 2023-07-20T01:43:00.282091+0000 (oldest deadline 2023-07-20T01:43:22.582158+0000) 2023-07-20T01:43:28.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:27 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:27.766+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:28.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:27 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:27.766+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:28.185 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:27 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:27.798+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:28.185 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:27 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:27.798+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:28.186 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:27 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:27.735+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:28.186 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:27 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:27.735+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:28.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:27 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:27.984+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:28.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:27 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:27.984+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:28.968 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:28 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:28.728+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:29.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:28 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:28.883+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:29.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:28 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:28.883+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:29.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:28 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:28.815+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:29.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:28 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:28.815+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:29.102 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:28 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:28.734+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:29.103 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:28 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:28.734+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.282244+0000 front 2023-07-20T01:43:00.282091+0000 (oldest deadline 2023-07-20T01:43:22.582158+0000) 2023-07-20T01:43:29.185 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:28 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:28.843+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:29.186 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:28 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:28.843+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:29.186 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:28 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:28.726+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:29.186 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:28 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:28.726+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:29.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:29 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:29.001+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:29.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:29 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:29.001+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:29.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:29 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:29.693+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:29.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:29 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:29.693+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:29.936 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:29 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:29.832+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:29.936 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:29 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:29.832+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:29.937 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:29 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:29.832+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.265319+0000 front 2023-07-20T01:43:07.265442+0000 (oldest deadline 2023-07-20T01:43:29.565271+0000) 2023-07-20T01:43:29.948 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:29 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:29.722+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:30.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:29 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:29.886+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:30.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:29 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:29.886+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:30.102 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:29 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:29.730+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:30.102 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:29 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:29.730+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.282244+0000 front 2023-07-20T01:43:00.282091+0000 (oldest deadline 2023-07-20T01:43:22.582158+0000) 2023-07-20T01:43:30.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:29 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:29.788+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:30.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:29 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:29.788+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:30.103 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:29 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:29.788+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:30.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:29 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:29.982+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:30.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:29 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:29.982+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:30.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:30 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:30.706+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:30.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:30 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:30.706+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:30.936 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:30 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:30.851+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:30.936 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:30 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:30.851+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:30.937 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:30 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:30.851+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.265319+0000 front 2023-07-20T01:43:07.265442+0000 (oldest deadline 2023-07-20T01:43:29.565271+0000) 2023-07-20T01:43:31.001 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:30 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:30.767+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:31.002 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:30 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:30.986+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:31.002 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:30 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:30.986+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:31.101 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:30 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:30.771+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.180326+0000 front 2023-07-20T01:42:52.180384+0000 (oldest deadline 2023-07-20T01:43:15.680503+0000) 2023-07-20T01:43:31.101 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:30 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5[109434]: 2023-07-20T01:43:30.771+0000 7f8a6cb50700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.282244+0000 front 2023-07-20T01:43:00.282091+0000 (oldest deadline 2023-07-20T01:43:22.582158+0000) 2023-07-20T01:43:31.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:30 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:30.828+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:31.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:30 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:30.828+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:31.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:30 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:30.828+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:31.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:30 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:30.897+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:31.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:30 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:30.897+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:31.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:30 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:30.897+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:31.833 INFO:journalctl@ceph.osd.5.smithi086.stdout:Jul 20 01:43:31 smithi086 bash[126446]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-5 2023-07-20T01:43:31.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:31 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:31.669+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:31.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:31 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:31.669+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:31.936 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:31 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:31.868+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:31.936 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:31 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:31.868+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:31.936 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:31 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:31.868+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.265319+0000 front 2023-07-20T01:43:07.265442+0000 (oldest deadline 2023-07-20T01:43:29.565271+0000) 2023-07-20T01:43:32.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:31 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:31.866+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:32.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:31 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:31.866+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:32.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:31 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:31.866+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:32.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:31 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:31.880+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:32.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:31 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:31.880+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:32.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:31 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:31.880+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:32.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:31 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:31.812+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:32.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:32 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:32.010+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:32.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:32 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:32.010+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:32.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:32 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:32.860+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:32.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:32 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:32.860+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:32.936 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:32 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:32.860+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.265319+0000 front 2023-07-20T01:43:07.265442+0000 (oldest deadline 2023-07-20T01:43:29.565271+0000) 2023-07-20T01:43:32.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:32 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:32.668+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:32.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:32 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:32.668+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:32.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:32 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:32.668+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:33.016 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:32 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:32.920+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:33.017 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:32 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:32.920+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:33.017 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:32 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:32.920+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:33.017 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:32 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:32.893+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:33.017 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:32 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:32.893+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:33.017 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:32 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:32.893+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:33.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.5.service' 2023-07-20T01:43:33.064 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:43:33.065 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-20T01:43:33.065 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-20T01:43:33.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.6 2023-07-20T01:43:33.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:32 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:32.848+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:33.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:32 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:32.982+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:33.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:32 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:32.982+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:33.432 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:33 smithi003 systemd[1]: Stopping Ceph osd.6 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:43:33.685 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:33 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:33.466+0000 7fca180de700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:43:33.686 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:33 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:33.466+0000 7fca180de700 -1 osd.6 80 *** Got signal Terminated *** 2023-07-20T01:43:33.686 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:33 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:33.466+0000 7fca180de700 -1 osd.6 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:43:33.686 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:33 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:33.676+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:33.686 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:33 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:33.676+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:33.686 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:33 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:33.676+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:34.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:33 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:33.965+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:34.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:33 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:33.965+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:34.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:33 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:33.965+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:34.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:33 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:33.874+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:34.103 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:33 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:33.874+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:34.103 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:33 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:33.874+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:34.185 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:33 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:33.845+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:34.186 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:33 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:33.845+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:34.186 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:33 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:33.845+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.265319+0000 front 2023-07-20T01:43:07.265442+0000 (oldest deadline 2023-07-20T01:43:29.565271+0000) 2023-07-20T01:43:34.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:33 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:33.865+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:34.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:33 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:33.865+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:34.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:33 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:33.991+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:34.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:33 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:33.991+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:34.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:33 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:33.991+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:34.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:34 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:34.805+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:34.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:34 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:34.805+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:34.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:34 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:34.805+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.265319+0000 front 2023-07-20T01:43:07.265442+0000 (oldest deadline 2023-07-20T01:43:29.565271+0000) 2023-07-20T01:43:34.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:34 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:34.649+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:34.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:34 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:34.649+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:34.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:34 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:34.649+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:35.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:34 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:34.968+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:35.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:34 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:34.968+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:35.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:34 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:34.968+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:35.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:34 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:34.843+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:35.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:34 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:34.843+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:35.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:34 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:34.843+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:35.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:34 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:34.888+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:35.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:34 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:34.888+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:35.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:34 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:34.997+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:35.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:34 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:34.997+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:35.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:34 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:34.997+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:35.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:35 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:35.829+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:35.936 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:35 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:35.829+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:35.936 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:35 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:35.829+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.265319+0000 front 2023-07-20T01:43:07.265442+0000 (oldest deadline 2023-07-20T01:43:29.565271+0000) 2023-07-20T01:43:35.937 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:35 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:35.694+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:35.937 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:35 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:35.694+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:35.937 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:35 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:35.694+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:36.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:35 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:35.876+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:36.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:35 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:35.876+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:36.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:35 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:35.990+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:36.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:35 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:35.990+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:36.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:35 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:35.990+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:36.351 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:35 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:35.952+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:36.352 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:35 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:35.952+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:36.352 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:35 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:35.952+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:36.353 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:35 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:35.892+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:36.353 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:35 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:35.892+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:36.353 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:35 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:35.892+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:36.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:36 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:36.879+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:36.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:36 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:36.879+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:36.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:36 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:36.879+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.265319+0000 front 2023-07-20T01:43:07.265442+0000 (oldest deadline 2023-07-20T01:43:29.565271+0000) 2023-07-20T01:43:36.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:36 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:36.675+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:36.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:36 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:36.675+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:36.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:36 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:36.675+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:37.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:36 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:36.994+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:37.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:36 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:36.994+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:37.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:36 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:36.994+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:37.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:36 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:36.861+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:37.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:36 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:36.861+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:37.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:36 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:36.861+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:37.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:36 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:36.847+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:37.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:36 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:36.847+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:37.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:37 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:37.024+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:37.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:37 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:37.024+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:37.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:37 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:37.024+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:37.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:37 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:37.024+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:37.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:37 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:37.851+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:53.763526+0000 front 2023-07-20T01:42:53.763500+0000 (oldest deadline 2023-07-20T01:43:18.463681+0000) 2023-07-20T01:43:37.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:37 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:37.851+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:58.464455+0000 front 2023-07-20T01:42:58.464275+0000 (oldest deadline 2023-07-20T01:43:24.364149+0000) 2023-07-20T01:43:37.935 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:37 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6[120496]: 2023-07-20T01:43:37.851+0000 7fca116cf700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.265319+0000 front 2023-07-20T01:43:07.265442+0000 (oldest deadline 2023-07-20T01:43:29.565271+0000) 2023-07-20T01:43:37.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:37 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:37.714+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:37.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:37 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:37.714+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:37.937 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:37 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:37.714+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:37.937 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:37 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:37.714+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:38.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:37.830+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:38.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:37.830+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:38.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:37.830+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:38.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:37.971+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:38.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:37.971+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:38.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:37 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:37.971+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:38.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:37 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:37.837+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:38.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:37 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:37.837+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:38.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:37 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:37.837+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.268204+0000 front 2023-07-20T01:43:15.268232+0000 (oldest deadline 2023-07-20T01:43:36.968106+0000) 2023-07-20T01:43:38.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:38 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:38.048+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:38.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:38 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:38.048+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:38.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:38 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:38.048+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:38.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:38 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:38.048+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:39.054 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:38 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:38.749+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:39.055 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:38 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:38.749+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:39.055 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:38 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:38.749+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:39.055 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:38 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:38.749+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:39.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:38 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:38.852+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:39.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:38 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:38.852+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:39.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:38 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:38.852+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:39.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:38 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:38.922+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:39.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:38 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:38.922+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:39.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:38 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:38.922+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:39.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:38 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:38.922+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:39.250 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:38 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:38.844+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:39.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:38 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:38.844+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:39.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:38 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:38.844+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.268204+0000 front 2023-07-20T01:43:15.268232+0000 (oldest deadline 2023-07-20T01:43:36.968106+0000) 2023-07-20T01:43:39.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:39.035+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:39.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:39.035+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:39.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:39.035+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:39.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:39.035+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:39.435 INFO:journalctl@ceph.osd.6.smithi003.stdout:Jul 20 01:43:39 smithi003 bash[134334]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-6 2023-07-20T01:43:39.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:39 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:39.700+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:39.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:39 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:39.700+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:39.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:39 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:39.700+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:39.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:39 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:39.700+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:40.104 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:39 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:39.867+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:40.105 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:39 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:39.867+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:40.105 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:39 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:39.867+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:40.105 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:39 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:39.879+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:40.106 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:39 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:39.879+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:40.106 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:39 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:39.879+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:40.106 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:39 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:39.879+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:40.250 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:39.801+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:40.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:39.801+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:40.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:39 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:39.801+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.268204+0000 front 2023-07-20T01:43:15.268232+0000 (oldest deadline 2023-07-20T01:43:36.968106+0000) 2023-07-20T01:43:40.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:40.026+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:40.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:40.026+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:40.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:40.026+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:40.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:40.026+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:40.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.6.service' 2023-07-20T01:43:40.535 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:43:40.536 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-20T01:43:40.536 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-20T01:43:40.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.7 2023-07-20T01:43:40.905 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:40 smithi043 systemd[1]: Stopping Ceph osd.7 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:43:40.905 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:40.817+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:40.905 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:40.817+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:40.906 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:40.817+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.268204+0000 front 2023-07-20T01:43:15.268232+0000 (oldest deadline 2023-07-20T01:43:36.968106+0000) 2023-07-20T01:43:40.906 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:40.938+0000 7fa3eb3ba700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:43:40.906 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:40.938+0000 7fa3eb3ba700 -1 osd.7 80 *** Got signal Terminated *** 2023-07-20T01:43:40.906 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:40 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:40.938+0000 7fa3eb3ba700 -1 osd.7 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:43:40.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:40 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:40.717+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:40.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:40 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:40.717+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:40.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:40 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:40.717+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:40.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:40 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:40.717+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:41.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:40.844+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:41.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:40.844+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:41.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:40.844+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:41.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:40.890+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:41.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:40.890+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:41.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:40.890+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:41.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:40 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:40.890+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:41.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:41 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:41.058+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:41.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:41 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:41.058+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:41.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:41 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:41.058+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:41.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:41 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:41.058+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:42.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:41 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:41.908+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:42.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:41 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:41.908+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:42.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:41 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:41.908+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:42.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:41 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:41.908+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:42.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:41 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:41.848+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:42.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:41 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:41.848+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:42.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:41 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:41.848+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:42.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:41 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:41.746+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:42.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:41 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:41.746+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:42.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:41 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:41.746+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:42.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:41 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:41.746+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:42.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:41 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:41.810+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:42.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:41 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:41.810+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:42.251 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:41 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:41.810+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.268204+0000 front 2023-07-20T01:43:15.268232+0000 (oldest deadline 2023-07-20T01:43:36.968106+0000) 2023-07-20T01:43:42.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:42.030+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:42.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:42.030+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:42.255 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:42.030+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:42.255 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:42.030+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:43.000 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:42.771+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:43.001 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:42.771+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:43.001 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:42.771+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.268204+0000 front 2023-07-20T01:43:15.268232+0000 (oldest deadline 2023-07-20T01:43:36.968106+0000) 2023-07-20T01:43:43.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:42.983+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:43.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:42.983+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:43.002 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:42.983+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:43.002 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:42 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:42.983+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:43.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:42.823+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:43.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:42.823+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:43.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:42.823+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:43.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:42.933+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:43.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:42.933+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:43.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:42.933+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:43.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:42 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:42.933+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:43.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:42 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:42.720+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:43.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:42 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:42.720+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:43.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:42 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:42.720+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:43.186 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:42 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:42.720+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:43.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:43 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:43.684+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:43.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:43 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:43.684+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:43.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:43 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:43.684+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:43.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:43 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:43.684+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:43.954 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:43.726+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:43.954 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:43.726+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:43.954 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:43.726+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.268204+0000 front 2023-07-20T01:43:15.268232+0000 (oldest deadline 2023-07-20T01:43:36.968106+0000) 2023-07-20T01:43:44.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:43 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:43.977+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:44.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:43 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:43.977+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:44.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:43 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:43.977+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:44.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:43 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:43.977+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:44.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:43 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:43.827+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:44.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:43 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:43.827+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:44.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:43 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:43.827+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:44.250 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:43.988+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:44.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:43.988+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:44.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:43.988+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:44.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:43 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:43.988+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:44.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:44 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:44.705+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:44.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:44 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:44.705+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:44.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:44 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:44.705+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:44.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:44 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:44.705+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:44.993 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:44 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:44.714+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:44.993 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:44 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:44.714+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:44.994 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:44 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:44.714+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.268204+0000 front 2023-07-20T01:43:15.268232+0000 (oldest deadline 2023-07-20T01:43:36.968106+0000) 2023-07-20T01:43:45.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:44 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:44.791+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:45.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:44 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:44.791+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:45.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:44 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:44.791+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:45.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:45 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:45.009+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:45.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:45 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:45.009+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:45.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:45 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:45.009+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:45.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:45 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:45.009+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:45.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:45.026+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:45.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:45.026+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:45.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:45.026+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:45.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:45.026+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:45.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:45.026+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:45.944 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:45.726+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.064363+0000 front 2023-07-20T01:42:59.064331+0000 (oldest deadline 2023-07-20T01:43:21.964585+0000) 2023-07-20T01:43:45.944 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:45.726+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:03.665585+0000 front 2023-07-20T01:43:03.665559+0000 (oldest deadline 2023-07-20T01:43:33.066490+0000) 2023-07-20T01:43:45.945 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7[113957]: 2023-07-20T01:43:45.726+0000 7fa3e59ad700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.268204+0000 front 2023-07-20T01:43:15.268232+0000 (oldest deadline 2023-07-20T01:43:36.968106+0000) 2023-07-20T01:43:46.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:46 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:46.019+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:46.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:46 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:46.019+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:46.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:46 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:46.019+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:46.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:46 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:46.019+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:46.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:45 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:45.797+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:46.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:45 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:45.797+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:46.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:45 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:45.797+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:46.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:45 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:45.797+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:46.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:45 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:45.727+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:46.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:45 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:45.727+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:46.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:45 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:45.727+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:46.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:45 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:45.727+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:46.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:45.978+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:46.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:45.978+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:46.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:45.978+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:46.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:45.978+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:46.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:45 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:45.978+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:46.582 INFO:journalctl@ceph.osd.7.smithi043.stdout:Jul 20 01:43:46 smithi043 bash[125399]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-7 2023-07-20T01:43:47.031 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:46 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:46.763+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:47.031 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:46 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:46.763+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:47.031 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:46 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:46.763+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:47.031 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:46 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:46.763+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:47.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:46 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:46.768+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:47.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:46 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:46.768+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:47.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:46 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:46.768+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:47.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:46 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:46.768+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:47.250 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:46 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:46.997+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:47.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:46 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:46.997+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:47.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:46 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:46.998+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:47.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:46 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:46.998+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:47.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:46 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:46.998+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:47.351 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:47 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:47.065+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:47.351 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:47 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:47.065+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:47.351 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:47 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:47.065+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:47.351 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:47 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:47.065+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:47.352 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:47 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:47.065+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.509168+0000 front 2023-07-20T01:43:20.509083+0000 (oldest deadline 2023-07-20T01:43:46.309755+0000) 2023-07-20T01:43:48.060 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.7.service' 2023-07-20T01:43:48.101 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:48.047+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:48.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:48.047+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:48.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:48.047+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:48.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:48.047+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:48.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:48.047+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.509168+0000 front 2023-07-20T01:43:20.509083+0000 (oldest deadline 2023-07-20T01:43:46.309755+0000) 2023-07-20T01:43:48.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:47 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:47.810+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:48.103 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:47 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:47.810+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:48.103 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:47 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:47.810+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:48.103 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:47 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:47.810+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:48.104 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:43:48.105 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-20T01:43:48.105 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-20T01:43:48.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.8 2023-07-20T01:43:48.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:47 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:47.719+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:48.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:47 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:47.719+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:48.185 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:47 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:47.719+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:48.186 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:47 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:47.719+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:48.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:47 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:47.985+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:48.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:47 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:47.985+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:48.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:47 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:47.985+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:48.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:47 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:47.985+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:48.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:47 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:47.985+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:48.468 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:48 smithi086 systemd[1]: Stopping Ceph osd.8 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:43:48.735 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:48.502+0000 7f6bb7222700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:43:48.735 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:48.502+0000 7f6bb7222700 -1 osd.8 80 *** Got signal Terminated *** 2023-07-20T01:43:48.735 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:48.502+0000 7f6bb7222700 -1 osd.8 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:43:48.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:48 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:48.697+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:48.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:48 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:48.697+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:48.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:48 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:48.697+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:48.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:48 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:48.697+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:49.007 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:48.769+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:49.007 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:48.769+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:49.007 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:48.769+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:49.007 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:48 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:48.769+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:49.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:48 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:48.959+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:49.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:48 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:48.959+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:49.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:48 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:48.959+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:49.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:48 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:48.959+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:49.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:48 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:48.959+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:49.351 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:49 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:49.041+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:49.351 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:49 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:49.041+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:49.351 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:49 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:49.041+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:49.351 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:49 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:49.041+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:49.352 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:49 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:49.041+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.509168+0000 front 2023-07-20T01:43:20.509083+0000 (oldest deadline 2023-07-20T01:43:46.309755+0000) 2023-07-20T01:43:49.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:49 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:49.653+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:49.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:49 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:49.653+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:49.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:49 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:49.653+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:49.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:49 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:49.653+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:49.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:49 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:49.653+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:43:50.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:49 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:49.793+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:50.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:49 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:49.793+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:50.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:49 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:49.793+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:50.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:49 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:49.793+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:50.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:50.007+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:50.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:50.007+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:50.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:50.007+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:50.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:50.007+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:50.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:50.007+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.509168+0000 front 2023-07-20T01:43:20.509083+0000 (oldest deadline 2023-07-20T01:43:46.309755+0000) 2023-07-20T01:43:50.250 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:49 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:49.914+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:50.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:49 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:49.914+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:50.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:49 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:49.914+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:50.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:49 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:49.914+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:50.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:49 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:49.914+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:50.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:50 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:50.629+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:50.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:50 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:50.629+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:50.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:50 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:50.629+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:50.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:50 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:50.629+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:50.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:50 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:50.629+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:43:51.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:50.797+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:51.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:50.797+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:51.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:50.797+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:51.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:50.797+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:51.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:50.992+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:51.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:50.992+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:51.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:50.992+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:51.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:50.992+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:51.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:50 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:50.992+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.509168+0000 front 2023-07-20T01:43:20.509083+0000 (oldest deadline 2023-07-20T01:43:46.309755+0000) 2023-07-20T01:43:51.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:50 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:50.946+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:51.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:50 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:50.946+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:51.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:50 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:50.946+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:51.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:50 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:50.946+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:51.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:50 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:50.946+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:51.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:51 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:51.647+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:51.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:51 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:51.647+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:51.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:51 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:51.647+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:51.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:51 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:51.647+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:51.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:51 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:51.647+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:43:52.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:51 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:51.813+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:52.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:51 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:51.813+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:52.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:51 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:51.813+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:52.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:51 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:51.813+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:52.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:51 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:51.813+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:43:52.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:52.013+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:52.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:52.013+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:52.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:52.013+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:52.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:52.013+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:52.104 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:52.013+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.509168+0000 front 2023-07-20T01:43:20.509083+0000 (oldest deadline 2023-07-20T01:43:46.309755+0000) 2023-07-20T01:43:52.104 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:52.013+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:29.711137+0000 front 2023-07-20T01:43:29.711177+0000 (oldest deadline 2023-07-20T01:43:52.011180+0000) 2023-07-20T01:43:52.250 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:51 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:51.925+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:52.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:51 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:51.925+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:52.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:51 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:51.925+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:52.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:51 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:51.925+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:52.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:51 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:51.925+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:52.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:52 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:52.635+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:52.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:52 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:52.635+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:52.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:52 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:52.635+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:52.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:52 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:52.635+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:52.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:52 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:52.635+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:43:52.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:52 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:52.635+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:30.574608+0000 front 2023-07-20T01:43:30.574540+0000 (oldest deadline 2023-07-20T01:43:51.674552+0000) 2023-07-20T01:43:53.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:52.775+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:53.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:52.775+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:53.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:52.775+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:53.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:52.775+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:53.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:52 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:52.775+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:43:53.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:53 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:53.002+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.703497+0000 front 2023-07-20T01:42:52.703618+0000 (oldest deadline 2023-07-20T01:43:17.904105+0000) 2023-07-20T01:43:53.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:53 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:53.002+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.805444+0000 front 2023-07-20T01:43:00.805458+0000 (oldest deadline 2023-07-20T01:43:24.905490+0000) 2023-07-20T01:43:53.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:53 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:53.002+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.905875+0000 front 2023-07-20T01:43:04.905957+0000 (oldest deadline 2023-07-20T01:43:30.206036+0000) 2023-07-20T01:43:53.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:53 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:53.002+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:14.807332+0000 front 2023-07-20T01:43:14.807494+0000 (oldest deadline 2023-07-20T01:43:38.907454+0000) 2023-07-20T01:43:53.102 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:53 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:53.002+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.509168+0000 front 2023-07-20T01:43:20.509083+0000 (oldest deadline 2023-07-20T01:43:46.309755+0000) 2023-07-20T01:43:53.103 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:53 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8[114963]: 2023-07-20T01:43:53.002+0000 7f6bb2016700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:29.711137+0000 front 2023-07-20T01:43:29.711177+0000 (oldest deadline 2023-07-20T01:43:52.011180+0000) 2023-07-20T01:43:53.250 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:52 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:52.885+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:53.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:52 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:52.885+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:53.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:52 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:52.885+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:53.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:52 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:52.885+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:53.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:52 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:52.885+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:53.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:53 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:53.590+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:53.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:53 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:53.590+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:53.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:53 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:53.590+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:53.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:53 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:53.590+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:53.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:53 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:53.590+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:43:53.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:53 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:53.590+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:30.574608+0000 front 2023-07-20T01:43:30.574540+0000 (oldest deadline 2023-07-20T01:43:51.674552+0000) 2023-07-20T01:43:54.051 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:53 smithi086 bash[126708]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-8 2023-07-20T01:43:54.052 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:53 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:53.814+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:54.052 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:53 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:53.814+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:54.052 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:53 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:53.814+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:54.052 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:53 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:53.814+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:54.052 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:53 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:53.814+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:43:54.250 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:53 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:53.857+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:54.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:53 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:53.857+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:54.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:53 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:53.857+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:54.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:53 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:53.857+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:54.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:53 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:53.857+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:54.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:53 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:53.857+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:43:54.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:54 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:54.849+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:54.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:54 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:54.849+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:54.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:54 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:54.849+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:54.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:54 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:54.849+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:54.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:54 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:54.849+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:43:54.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:54 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:54.567+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:54.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:54 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:54.567+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:54.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:54 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:54.567+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:54.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:54 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:54.567+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:54.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:54 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:54.567+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:43:54.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:54 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:54.567+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:30.574608+0000 front 2023-07-20T01:43:30.574540+0000 (oldest deadline 2023-07-20T01:43:51.674552+0000) 2023-07-20T01:43:55.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.8.service' 2023-07-20T01:43:55.236 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:55 smithi086 systemd[1]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.8.service: Succeeded. 2023-07-20T01:43:55.237 INFO:journalctl@ceph.osd.8.smithi086.stdout:Jul 20 01:43:55 smithi086 systemd[1]: Stopped Ceph osd.8 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:43:55.250 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:43:55.250 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-20T01:43:55.251 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-20T01:43:55.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.9 2023-07-20T01:43:55.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:54 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:54.821+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:55.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:54 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:54.821+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:55.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:54 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:54.821+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:55.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:54 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:54.821+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:55.253 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:54 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:54.821+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:55.253 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:54 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:54.821+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:43:55.551 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:55 smithi003 systemd[1]: Stopping Ceph osd.9 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:43:55.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:55 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:55.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:55.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:55 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:55.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:55.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:55 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:55.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:55.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:55 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:55.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:55.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:55 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:55.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:43:55.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:55 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:55.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:30.574608+0000 front 2023-07-20T01:43:30.574540+0000 (oldest deadline 2023-07-20T01:43:51.674552+0000) 2023-07-20T01:43:55.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:55 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:55.678+0000 7f0647d1d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:43:55.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:55 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:55.678+0000 7f0647d1d700 -1 osd.9 80 *** Got signal Terminated *** 2023-07-20T01:43:55.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:55 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:55.678+0000 7f0647d1d700 -1 osd.9 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:43:56.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:55 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:55.842+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:56.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:55 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:55.842+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:56.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:55 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:55.842+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:56.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:55 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:55.842+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:56.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:55 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:55.842+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:43:56.250 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:55 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:55.791+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:56.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:55 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:55.791+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:56.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:55 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:55.791+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:56.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:55 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:55.791+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:56.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:55 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:55.791+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:56.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:55 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:55.791+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:43:56.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:56 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:56.631+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:56.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:56 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:56.631+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:56.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:56 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:56.631+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:56.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:56 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:56.631+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:56.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:56 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:56.631+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:43:56.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:56 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:56.631+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:30.574608+0000 front 2023-07-20T01:43:30.574540+0000 (oldest deadline 2023-07-20T01:43:51.674552+0000) 2023-07-20T01:43:57.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:56 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:56.841+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:57.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:56 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:56.841+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:57.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:56 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:56.841+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:57.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:56 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:56.841+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:57.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:56 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:56.841+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:43:57.250 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:56 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:56.802+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:57.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:56 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:56.802+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:57.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:56 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:56.802+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:57.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:56 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:56.802+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:57.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:56 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:56.802+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:57.252 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:56 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:56.802+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:43:57.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:57 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:57.605+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:57.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:57 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:57.606+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:57.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:57 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:57.606+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:57.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:57 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:57.606+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:57.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:57 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:57.606+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:43:57.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:57 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:57.606+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:30.574608+0000 front 2023-07-20T01:43:30.574540+0000 (oldest deadline 2023-07-20T01:43:51.674552+0000) 2023-07-20T01:43:58.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:57 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:57.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:58.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:57 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:57.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:58.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:57 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:57.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:58.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:57 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:57.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:58.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:57 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:57.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:58.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:57 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:57.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:43:58.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:57 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:57.868+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:58.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:57 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:57.868+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:58.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:57 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:57.868+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:58.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:57 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:57.868+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:58.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:57 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:57.868+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:43:58.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:58 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:58.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:58.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:58 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:58.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:58.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:58 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:58.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:58.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:58 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:58.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:58.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:58 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:58.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:43:58.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:58 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:58.584+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:30.574608+0000 front 2023-07-20T01:43:30.574540+0000 (oldest deadline 2023-07-20T01:43:51.674552+0000) 2023-07-20T01:43:59.000 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:58 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:58.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:43:59.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:58 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:58.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:43:59.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:58 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:58.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:43:59.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:58 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:58.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:43:59.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:58 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:58.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:43:59.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:58 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:58.775+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:43:59.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:58 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:58.819+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:43:59.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:58 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:58.819+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:43:59.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:58 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:58.819+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:43:59.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:58 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:58.819+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:43:59.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:58 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:58.819+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:43:59.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:59 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:59.600+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:43:59.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:59 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:59.600+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:43:59.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:59 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:59.600+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:43:59.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:59 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:59.600+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:43:59.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:59 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:59.600+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:43:59.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:43:59 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:43:59.600+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:30.574608+0000 front 2023-07-20T01:43:30.574540+0000 (oldest deadline 2023-07-20T01:43:51.674552+0000) 2023-07-20T01:44:00.000 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:59 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:59.734+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:44:00.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:59 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:59.734+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:44:00.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:59 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:59.734+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:44:00.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:59 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:59.734+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:44:00.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:59 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:59.734+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:44:00.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:59 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:59.734+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:44:00.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:43:59 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:43:59.734+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.868136+0000 front 2023-07-20T01:43:37.868190+0000 (oldest deadline 2023-07-20T01:43:59.568139+0000) 2023-07-20T01:44:00.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:59 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:59.770+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:00.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:59 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:59.770+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:00.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:59 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:59.770+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:00.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:59 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:59.770+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:00.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:43:59 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:43:59.770+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:00.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:44:00 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:44:00.597+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:48.368474+0000 front 2023-07-20T01:42:48.368507+0000 (oldest deadline 2023-07-20T01:43:20.169434+0000) 2023-07-20T01:44:00.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:44:00 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:44:00.597+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:43:00.169929+0000 front 2023-07-20T01:43:00.170048+0000 (oldest deadline 2023-07-20T01:43:26.069960+0000) 2023-07-20T01:44:00.935 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:44:00 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:44:00.597+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:06.070488+0000 front 2023-07-20T01:43:06.070515+0000 (oldest deadline 2023-07-20T01:43:31.970602+0000) 2023-07-20T01:44:00.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:44:00 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:44:00.597+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:15.471562+0000 front 2023-07-20T01:43:15.471616+0000 (oldest deadline 2023-07-20T01:43:37.171860+0000) 2023-07-20T01:44:00.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:44:00 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:44:00.597+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:17.672774+0000 front 2023-07-20T01:43:17.672958+0000 (oldest deadline 2023-07-20T01:43:48.873270+0000) 2023-07-20T01:44:00.936 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:44:00 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9[126009]: 2023-07-20T01:44:00.597+0000 7f0643b13700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:30.574608+0000 front 2023-07-20T01:43:30.574540+0000 (oldest deadline 2023-07-20T01:43:51.674552+0000) 2023-07-20T01:44:01.000 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:00 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:00.712+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:44:01.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:00 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:00.712+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:44:01.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:00 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:00.712+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:44:01.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:00 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:00.712+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:44:01.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:00 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:00.712+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:44:01.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:00 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:00.712+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:44:01.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:00 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:00.712+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.868136+0000 front 2023-07-20T01:43:37.868190+0000 (oldest deadline 2023-07-20T01:43:59.568139+0000) 2023-07-20T01:44:01.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:00 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:00.733+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:01.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:00 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:00.733+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:01.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:00 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:00.733+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:01.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:00 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:00.733+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:01.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:00 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:00.733+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:01.435 INFO:journalctl@ceph.osd.9.smithi003.stdout:Jul 20 01:44:01 smithi003 bash[134607]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-9 2023-07-20T01:44:02.000 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:01 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:01.684+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:44:02.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:01 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:01.684+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:44:02.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:01 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:01.684+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:44:02.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:01 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:01.684+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:44:02.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:01 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:01.684+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:44:02.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:01 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:01.684+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:44:02.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:01 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:01.684+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.868136+0000 front 2023-07-20T01:43:37.868190+0000 (oldest deadline 2023-07-20T01:43:59.568139+0000) 2023-07-20T01:44:02.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:01 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:01.699+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:02.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:01 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:01.699+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:02.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:01 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:01.699+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:02.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:01 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:01.699+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:02.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:01 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:01.699+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:02.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.9.service' 2023-07-20T01:44:02.617 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:44:02.617 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-20T01:44:02.618 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-20T01:44:02.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.10 2023-07-20T01:44:02.985 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:02 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:02.643+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:44:02.985 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:02 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:02.643+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:44:02.985 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:02 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:02.643+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:44:02.986 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:02 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:02.643+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:44:02.986 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:02 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:02.643+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:44:02.986 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:02 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:02.643+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:44:02.986 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:02 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:02.643+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.868136+0000 front 2023-07-20T01:43:37.868190+0000 (oldest deadline 2023-07-20T01:43:59.568139+0000) 2023-07-20T01:44:02.986 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:02 smithi043 systemd[1]: Stopping Ceph osd.10 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:44:02.986 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:03 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:03.018+0000 7f913b5f9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:44:03.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:02 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:02.725+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:03.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:02 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:02.725+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:03.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:02 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:02.725+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:03.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:02 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:02.725+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:03.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:02 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:02.725+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:03.250 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:03 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:03.018+0000 7f913b5f9700 -1 osd.10 80 *** Got signal Terminated *** 2023-07-20T01:44:03.251 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:03 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:03.018+0000 7f913b5f9700 -1 osd.10 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:44:04.000 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:03 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:03.618+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:44:04.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:03 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:03.618+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:44:04.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:03 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:03.618+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:44:04.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:03 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:03.618+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:44:04.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:03 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:03.618+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:44:04.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:03 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:03.618+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:44:04.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:03 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:03.618+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.868136+0000 front 2023-07-20T01:43:37.868190+0000 (oldest deadline 2023-07-20T01:43:59.568139+0000) 2023-07-20T01:44:04.100 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:03 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:03.731+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:04.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:03 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:03.731+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:04.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:03 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:03.731+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:04.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:03 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:03.731+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:04.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:03 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:03.731+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:04.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:03 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:03.731+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:05.000 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:04 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:04.650+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:44:05.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:04 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:04.650+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:44:05.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:04 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:04.650+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:44:05.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:04 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:04.650+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:44:05.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:04 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:04.650+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:44:05.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:04 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:04.650+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:44:05.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:04 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:04.650+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.868136+0000 front 2023-07-20T01:43:37.868190+0000 (oldest deadline 2023-07-20T01:43:59.568139+0000) 2023-07-20T01:44:05.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:04 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:04.690+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:05.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:04 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:04.690+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:05.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:04 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:04.690+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:05.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:04 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:04.690+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:05.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:04 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:04.690+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:05.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:04 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:04.690+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:06.000 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:05 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:05.622+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:44:06.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:05 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:05.622+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:44:06.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:05 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:05.622+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:44:06.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:05 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:05.622+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:44:06.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:05 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:05.622+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:44:06.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:05 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:05.622+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:44:06.002 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:05 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:05.622+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.868136+0000 front 2023-07-20T01:43:37.868190+0000 (oldest deadline 2023-07-20T01:43:59.568139+0000) 2023-07-20T01:44:06.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:05 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:05.661+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:06.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:05 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:05.661+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:06.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:05 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:05.661+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:06.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:05 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:05.661+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:06.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:05 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:05.661+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:06.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:05 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:05.661+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:06.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:06 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:06.616+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:06.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:06 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:06.616+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:06.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:06 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:06.616+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:06.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:06 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:06.616+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:06.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:06 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:06.616+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:06.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:06 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:06.616+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:07.000 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:06 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:06.670+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:44:07.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:06 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:06.670+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:44:07.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:06 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:06.670+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:44:07.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:06 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:06.670+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:44:07.002 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:06 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:06.670+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:44:07.002 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:06 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:06.670+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:44:07.002 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:06 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:06.670+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.868136+0000 front 2023-07-20T01:43:37.868190+0000 (oldest deadline 2023-07-20T01:43:59.568139+0000) 2023-07-20T01:44:07.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:07 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:07.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:07.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:07 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:07.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:07.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:07 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:07.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:07.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:07 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:07.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:07.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:07 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:07.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:07.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:07 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:07.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:08.000 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:07 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:07.630+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:52.961295+0000 front 2023-07-20T01:42:52.961323+0000 (oldest deadline 2023-07-20T01:43:15.861328+0000) 2023-07-20T01:44:08.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:07 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:07.630+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.362351+0000 front 2023-07-20T01:42:59.362289+0000 (oldest deadline 2023-07-20T01:43:22.262477+0000) 2023-07-20T01:44:08.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:07 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:07.630+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:07.364804+0000 front 2023-07-20T01:43:07.364787+0000 (oldest deadline 2023-07-20T01:43:33.264766+0000) 2023-07-20T01:44:08.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:07 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:07.630+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-07-20T01:43:13.265365+0000 front 2023-07-20T01:43:13.265415+0000 (oldest deadline 2023-07-20T01:43:36.765239+0000) 2023-07-20T01:44:08.001 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:07 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:07.630+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:20.866225+0000 front 2023-07-20T01:43:20.866306+0000 (oldest deadline 2023-07-20T01:43:44.966357+0000) 2023-07-20T01:44:08.002 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:07 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:07.630+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:27.267411+0000 front 2023-07-20T01:43:27.267290+0000 (oldest deadline 2023-07-20T01:43:53.167268+0000) 2023-07-20T01:44:08.002 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:07 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10[119419]: 2023-07-20T01:44:07.630+0000 7f91343e9700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.868136+0000 front 2023-07-20T01:43:37.868190+0000 (oldest deadline 2023-07-20T01:43:59.568139+0000) 2023-07-20T01:44:08.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:08 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:08.587+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:08.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:08 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:08.587+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:08.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:08 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:08.587+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:08.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:08 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:08.587+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:08.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:08 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:08.587+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:08.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:08 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:08.587+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:09.000 INFO:journalctl@ceph.osd.10.smithi043.stdout:Jul 20 01:44:08 smithi043 bash[125671]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-10 2023-07-20T01:44:09.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:09 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:09.575+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:09.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:09 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:09.575+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:09.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:09 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:09.575+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:09.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:09 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:09.575+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:09.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:09 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:09.575+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:09.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:09 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:09.575+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:09.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:09 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:09.575+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6820 osd.7 since back 2023-07-20T01:43:43.293029+0000 front 2023-07-20T01:43:43.293068+0000 (oldest deadline 2023-07-20T01:44:09.093326+0000) 2023-07-20T01:44:10.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.10.service' 2023-07-20T01:44:10.490 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:44:10.490 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-20T01:44:10.491 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-20T01:44:10.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.11 2023-07-20T01:44:10.841 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:10 smithi086 systemd[1]: Stopping Ceph osd.11 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:44:10.842 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:10 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:10.597+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:10.842 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:10 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:10.597+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:10.842 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:10 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:10.597+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:10.843 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:10 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:10.597+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:10.843 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:10 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:10.597+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:10.844 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:10 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:10.597+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:10.844 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:10 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:10.597+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6820 osd.7 since back 2023-07-20T01:43:43.293029+0000 front 2023-07-20T01:43:43.293068+0000 (oldest deadline 2023-07-20T01:44:09.093326+0000) 2023-07-20T01:44:10.844 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:10 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:10.875+0000 7f3edabcb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:44:10.844 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:10 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:10.875+0000 7f3edabcb700 -1 osd.11 80 *** Got signal Terminated *** 2023-07-20T01:44:10.845 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:10 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:10.875+0000 7f3edabcb700 -1 osd.11 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:44:11.850 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:11 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:11.603+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:11.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:11 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:11.603+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:11.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:11 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:11.603+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:11.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:11 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:11.603+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:11.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:11 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:11.603+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:11.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:11 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:11.603+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:11.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:11 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:11.603+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6820 osd.7 since back 2023-07-20T01:43:43.293029+0000 front 2023-07-20T01:43:43.293068+0000 (oldest deadline 2023-07-20T01:44:09.093326+0000) 2023-07-20T01:44:12.850 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:12 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:12.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:12.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:12 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:12.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:12.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:12 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:12.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:12.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:12 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:12.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:12.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:12 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:12.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:12.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:12 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:12.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:12.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:12 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:12.609+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6820 osd.7 since back 2023-07-20T01:43:43.293029+0000 front 2023-07-20T01:43:43.293068+0000 (oldest deadline 2023-07-20T01:44:09.093326+0000) 2023-07-20T01:44:13.850 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:13 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:13.615+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:13.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:13 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:13.615+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:13.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:13 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:13.615+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:13.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:13 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:13.615+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:13.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:13 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:13.615+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:13.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:13 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:13.615+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:13.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:13 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:13.615+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6820 osd.7 since back 2023-07-20T01:43:43.293029+0000 front 2023-07-20T01:43:43.293068+0000 (oldest deadline 2023-07-20T01:44:09.093326+0000) 2023-07-20T01:44:14.850 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:14 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:14.626+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:14.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:14 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:14.626+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:14.851 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:14 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:14.626+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:14.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:14 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:14.626+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:14.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:14 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:14.626+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:14.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:14 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:14.626+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:14.852 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:14 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:14.626+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6820 osd.7 since back 2023-07-20T01:43:43.293029+0000 front 2023-07-20T01:43:43.293068+0000 (oldest deadline 2023-07-20T01:44:09.093326+0000) 2023-07-20T01:44:16.100 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:15 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:15.638+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-07-20T01:42:51.981925+0000 front 2023-07-20T01:42:51.982077+0000 (oldest deadline 2023-07-20T01:43:14.882074+0000) 2023-07-20T01:44:16.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:15 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:15.638+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6804 osd.1 since back 2023-07-20T01:42:59.583488+0000 front 2023-07-20T01:42:59.583367+0000 (oldest deadline 2023-07-20T01:43:21.883389+0000) 2023-07-20T01:44:16.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:15 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:15.638+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2023-07-20T01:43:04.184223+0000 front 2023-07-20T01:43:04.184324+0000 (oldest deadline 2023-07-20T01:43:29.484453+0000) 2023-07-20T01:44:16.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:15 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:15.638+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6812 osd.4 since back 2023-07-20T01:43:23.288214+0000 front 2023-07-20T01:43:23.288322+0000 (oldest deadline 2023-07-20T01:43:44.888878+0000) 2023-07-20T01:44:16.101 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:15 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:15.638+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2023-07-20T01:43:26.490266+0000 front 2023-07-20T01:43:26.490325+0000 (oldest deadline 2023-07-20T01:43:51.790386+0000) 2023-07-20T01:44:16.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:15 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:15.638+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-07-20T01:43:37.392531+0000 front 2023-07-20T01:43:37.392571+0000 (oldest deadline 2023-07-20T01:44:03.292461+0000) 2023-07-20T01:44:16.102 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:15 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11[120509]: 2023-07-20T01:44:15.638+0000 7f3ed51be700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.43:6820 osd.7 since back 2023-07-20T01:43:43.293029+0000 front 2023-07-20T01:43:43.293068+0000 (oldest deadline 2023-07-20T01:44:09.093326+0000) 2023-07-20T01:44:16.850 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:16 smithi086 bash[126989]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-osd-11 2023-07-20T01:44:17.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.11.service' 2023-07-20T01:44:17.692 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:17 smithi086 systemd[1]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@osd.11.service: Succeeded. 2023-07-20T01:44:17.692 INFO:journalctl@ceph.osd.11.smithi086.stdout:Jul 20 01:44:17 smithi086 systemd[1]: Stopped Ceph osd.11 for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:44:17.705 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:44:17.706 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-20T01:44:17.706 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-20T01:44:17.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.a 2023-07-20T01:44:18.085 INFO:journalctl@ceph.mds.a.smithi003.stdout:Jul 20 01:44:17 smithi003 systemd[1]: Stopping Ceph mds.a for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:44:18.434 INFO:journalctl@ceph.mds.a.smithi003.stdout:Jul 20 01:44:18 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-a[131317]: 2023-07-20T01:44:18.119+0000 7ffb15a63700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:44:18.435 INFO:journalctl@ceph.mds.a.smithi003.stdout:Jul 20 01:44:18 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-a[131317]: 2023-07-20T01:44:18.119+0000 7ffb15a63700 -1 mds.a *** got signal Terminated *** 2023-07-20T01:44:22.684 INFO:journalctl@ceph.mds.a.smithi003.stdout:Jul 20 01:44:22 smithi003 bash[134869]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-a 2023-07-20T01:44:22.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.a.service' 2023-07-20T01:44:22.983 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:44:22.984 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-20T01:44:22.984 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-20T01:44:22.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.d 2023-07-20T01:44:23.375 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:44:23 smithi003 systemd[1]: Stopping Ceph mds.d for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:44:23.684 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:44:23 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-d[130315]: 2023-07-20T01:44:23.408+0000 7f7f1ff5a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:44:23.685 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:44:23 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-d[130315]: 2023-07-20T01:44:23.408+0000 7f7f1ff5a700 -1 mds.d *** got signal Terminated *** 2023-07-20T01:44:27.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.d.service' 2023-07-20T01:44:27.362 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:44:27 smithi003 bash[135013]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-d 2023-07-20T01:44:27.362 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:44:27 smithi003 systemd[1]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.d.service: Succeeded. 2023-07-20T01:44:27.363 INFO:journalctl@ceph.mds.d.smithi003.stdout:Jul 20 01:44:27 smithi003 systemd[1]: Stopped Ceph mds.d for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:44:27.374 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:44:27.374 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-20T01:44:27.375 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-20T01:44:27.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.g 2023-07-20T01:44:27.684 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:44:27 smithi003 systemd[1]: Stopping Ceph mds.g for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:44:28.184 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:44:27 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-g[130929]: 2023-07-20T01:44:27.789+0000 7f98b046b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:44:28.185 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:44:27 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-g[130929]: 2023-07-20T01:44:27.789+0000 7f98b046b700 -1 mds.g *** got signal Terminated *** 2023-07-20T01:44:32.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.g.service' 2023-07-20T01:44:32.555 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:44:32 smithi003 bash[135151]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-g 2023-07-20T01:44:32.556 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:44:32 smithi003 systemd[1]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.g.service: Succeeded. 2023-07-20T01:44:32.556 INFO:journalctl@ceph.mds.g.smithi003.stdout:Jul 20 01:44:32 smithi003 systemd[1]: Stopped Ceph mds.g for 9b4fd2fc-269d-11ee-9b34-001a4aab830c. 2023-07-20T01:44:32.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:44:32.566 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-20T01:44:32.567 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-20T01:44:32.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.j 2023-07-20T01:44:32.934 INFO:journalctl@ceph.mds.j.smithi003.stdout:Jul 20 01:44:32 smithi003 systemd[1]: Stopping Ceph mds.j for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:44:33.434 INFO:journalctl@ceph.mds.j.smithi003.stdout:Jul 20 01:44:32 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-j[129929]: 2023-07-20T01:44:32.988+0000 7f3b5fcb9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:44:33.435 INFO:journalctl@ceph.mds.j.smithi003.stdout:Jul 20 01:44:32 smithi003 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-j[129929]: 2023-07-20T01:44:32.988+0000 7f3b5fcb9700 -1 mds.j *** got signal Terminated *** 2023-07-20T01:44:36.920 INFO:journalctl@ceph.mds.j.smithi003.stdout:Jul 20 01:44:36 smithi003 bash[135291]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-j 2023-07-20T01:44:37.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.j.service' 2023-07-20T01:44:37.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:44:37.168 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-20T01:44:37.169 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-20T01:44:37.169 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.b 2023-07-20T01:44:37.500 INFO:journalctl@ceph.mds.b.smithi043.stdout:Jul 20 01:44:37 smithi043 systemd[1]: Stopping Ceph mds.b for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:44:38.000 INFO:journalctl@ceph.mds.b.smithi043.stdout:Jul 20 01:44:37 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-b[123102]: 2023-07-20T01:44:37.571+0000 7f7d6f0d6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:44:38.000 INFO:journalctl@ceph.mds.b.smithi043.stdout:Jul 20 01:44:37 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-b[123102]: 2023-07-20T01:44:37.571+0000 7f7d6f0d6700 -1 mds.b *** got signal Terminated *** 2023-07-20T01:44:44.000 INFO:journalctl@ceph.mds.b.smithi043.stdout:Jul 20 01:44:43 smithi043 bash[125941]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-b 2023-07-20T01:44:44.116 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.b.service' 2023-07-20T01:44:44.153 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:44:44.154 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-20T01:44:44.154 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-20T01:44:44.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.e 2023-07-20T01:44:44.500 INFO:journalctl@ceph.mds.e.smithi043.stdout:Jul 20 01:44:44 smithi043 systemd[1]: Stopping Ceph mds.e for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:44:45.000 INFO:journalctl@ceph.mds.e.smithi043.stdout:Jul 20 01:44:44 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-e[122424]: 2023-07-20T01:44:44.558+0000 7f0f231c4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:44:45.000 INFO:journalctl@ceph.mds.e.smithi043.stdout:Jul 20 01:44:44 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-e[122424]: 2023-07-20T01:44:44.558+0000 7f0f231c4700 -1 mds.e *** got signal Terminated *** 2023-07-20T01:44:49.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.e.service' 2023-07-20T01:44:49.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:44:49.150 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-20T01:44:49.150 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-20T01:44:49.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.h 2023-07-20T01:44:49.500 INFO:journalctl@ceph.mds.h.smithi043.stdout:Jul 20 01:44:49 smithi043 systemd[1]: Stopping Ceph mds.h for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:44:50.000 INFO:journalctl@ceph.mds.h.smithi043.stdout:Jul 20 01:44:49 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-h[122762]: 2023-07-20T01:44:49.578+0000 7f2d7c5c5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:44:50.000 INFO:journalctl@ceph.mds.h.smithi043.stdout:Jul 20 01:44:49 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-h[122762]: 2023-07-20T01:44:49.578+0000 7f2d7c5c5700 -1 mds.h *** got signal Terminated *** 2023-07-20T01:44:54.199 INFO:journalctl@ceph.mds.h.smithi043.stdout:Jul 20 01:44:54 smithi043 bash[126223]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-h 2023-07-20T01:44:54.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.h.service' 2023-07-20T01:44:54.246 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:44:54.247 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-20T01:44:54.248 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-20T01:44:54.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.k 2023-07-20T01:44:54.500 INFO:journalctl@ceph.mds.k.smithi043.stdout:Jul 20 01:44:54 smithi043 systemd[1]: Stopping Ceph mds.k for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:44:55.000 INFO:journalctl@ceph.mds.k.smithi043.stdout:Jul 20 01:44:54 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-k[122085]: 2023-07-20T01:44:54.660+0000 7fd4fc7c1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:44:55.000 INFO:journalctl@ceph.mds.k.smithi043.stdout:Jul 20 01:44:54 smithi043 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-k[122085]: 2023-07-20T01:44:54.660+0000 7fd4fc7c1700 -1 mds.k *** got signal Terminated *** 2023-07-20T01:44:59.000 INFO:journalctl@ceph.mds.k.smithi043.stdout:Jul 20 01:44:58 smithi043 bash[126371]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-k 2023-07-20T01:44:59.012 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.k.service' 2023-07-20T01:44:59.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:44:59.047 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-20T01:44:59.048 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-20T01:44:59.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.c 2023-07-20T01:44:59.350 INFO:journalctl@ceph.mds.c.smithi086.stdout:Jul 20 01:44:59 smithi086 systemd[1]: Stopping Ceph mds.c for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:44:59.850 INFO:journalctl@ceph.mds.c.smithi086.stdout:Jul 20 01:44:59 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-c[124108]: 2023-07-20T01:44:59.435+0000 7f7927040700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:44:59.850 INFO:journalctl@ceph.mds.c.smithi086.stdout:Jul 20 01:44:59 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-c[124108]: 2023-07-20T01:44:59.435+0000 7f7927040700 -1 mds.c *** got signal Terminated *** 2023-07-20T01:45:05.095 INFO:journalctl@ceph.mds.c.smithi086.stdout:Jul 20 01:45:04 smithi086 bash[127266]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-c 2023-07-20T01:45:05.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.c.service' 2023-07-20T01:45:05.139 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:45:05.140 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-20T01:45:05.140 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-20T01:45:05.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.f 2023-07-20T01:45:05.350 INFO:journalctl@ceph.mds.f.smithi086.stdout:Jul 20 01:45:05 smithi086 systemd[1]: Stopping Ceph mds.f for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:45:05.850 INFO:journalctl@ceph.mds.f.smithi086.stdout:Jul 20 01:45:05 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-f[123767]: 2023-07-20T01:45:05.543+0000 7fccb9bfb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:45:05.850 INFO:journalctl@ceph.mds.f.smithi086.stdout:Jul 20 01:45:05 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-f[123767]: 2023-07-20T01:45:05.543+0000 7fccb9bfb700 -1 mds.f *** got signal Terminated *** 2023-07-20T01:45:11.094 INFO:journalctl@ceph.mds.f.smithi086.stdout:Jul 20 01:45:10 smithi086 bash[127418]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-f 2023-07-20T01:45:11.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.f.service' 2023-07-20T01:45:11.136 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:45:11.136 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-20T01:45:11.137 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-20T01:45:11.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.i 2023-07-20T01:45:11.350 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:45:11 smithi086 systemd[1]: Stopping Ceph mds.i for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:45:11.850 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:45:11 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-i[124445]: 2023-07-20T01:45:11.542+0000 7f1005dcf700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:45:11.850 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:45:11 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-i[124445]: 2023-07-20T01:45:11.542+0000 7f1005dcf700 -1 mds.i *** got signal Terminated *** 2023-07-20T01:45:21.598 INFO:journalctl@ceph.mds.i.smithi086.stdout:Jul 20 01:45:21 smithi086 bash[127559]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-i 2023-07-20T01:45:21.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.i.service' 2023-07-20T01:45:21.637 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:45:21.638 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-20T01:45:21.639 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-20T01:45:21.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.l 2023-07-20T01:45:22.006 INFO:journalctl@ceph.mds.l.smithi086.stdout:Jul 20 01:45:21 smithi086 systemd[1]: Stopping Ceph mds.l for 9b4fd2fc-269d-11ee-9b34-001a4aab830c... 2023-07-20T01:45:22.350 INFO:journalctl@ceph.mds.l.smithi086.stdout:Jul 20 01:45:22 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-l[123427]: 2023-07-20T01:45:22.041+0000 7f28376b2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:45:22.350 INFO:journalctl@ceph.mds.l.smithi086.stdout:Jul 20 01:45:22 smithi086 ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-l[123427]: 2023-07-20T01:45:22.041+0000 7f28376b2700 -1 mds.l *** got signal Terminated *** 2023-07-20T01:45:24.561 INFO:journalctl@ceph.mds.l.smithi086.stdout:Jul 20 01:45:24 smithi086 bash[127699]: ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c-mds-l 2023-07-20T01:45:24.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b4fd2fc-269d-11ee-9b34-001a4aab830c@mds.l.service' 2023-07-20T01:45:24.600 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:45:24.601 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-20T01:45:24.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T01:45:24.920 INFO:teuthology.orchestra.run.smithi003.stdout:Deleting cluster with fsid: 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:45:26.575 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T01:45:26.898 INFO:teuthology.orchestra.run.smithi043.stdout:Deleting cluster with fsid: 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:45:28.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T01:45:28.870 INFO:teuthology.orchestra.run.smithi086.stdout:Deleting cluster with fsid: 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:45:30.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:45:30.461 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:45:30.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:45:30.529 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-20T01:45:30.531 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251/remote/smithi003/crash 2023-07-20T01:45:30.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T01:45:30.578 INFO:teuthology.orchestra.run.smithi003.stderr:tar: /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T01:45:30.578 INFO:teuthology.orchestra.run.smithi003.stderr:tar: Error is not recoverable: exiting now 2023-07-20T01:45:30.580 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251/remote/smithi043/crash 2023-07-20T01:45:30.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T01:45:30.626 INFO:teuthology.orchestra.run.smithi043.stderr:tar: /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T01:45:30.626 INFO:teuthology.orchestra.run.smithi043.stderr:tar: Error is not recoverable: exiting now 2023-07-20T01:45:30.628 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251/remote/smithi086/crash 2023-07-20T01:45:30.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T01:45:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/lib/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T01:45:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2023-07-20T01:45:30.668 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-20T01:45:30.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-07-20T01:45:30.705 INFO:teuthology.orchestra.run.smithi003.stderr:grep: /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph.log: No such file or directory 2023-07-20T01:45:30.707 INFO:tasks.cephadm:Compressing logs... 2023-07-20T01:45:30.708 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T01:45:30.750 DEBUG:teuthology.orchestra.run.smithi043:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T01:45:30.753 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T01:45:30.785 INFO:teuthology.orchestra.run.smithi003.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T01:45:30.785 INFO:teuthology.orchestra.run.smithi003.stderr:: No such file or directory 2023-07-20T01:45:30.785 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.0.log 2023-07-20T01:45:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.g.log 2023-07-20T01:45:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.a.log 2023-07-20T01:45:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.0.log: /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.6.log 2023-07-20T01:45:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.3.log 2023-07-20T01:45:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.d.log 2023-07-20T01:45:30.788 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T01:45:30.788 INFO:teuthology.orchestra.run.smithi086.stderr:: No such file or directory 2023-07-20T01:45:30.788 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.5.log 2023-07-20T01:45:30.789 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.2.log 2023-07-20T01:45:30.789 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.11.log 2023-07-20T01:45:30.789 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.5.log: /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mgr.z.log 2023-07-20T01:45:30.789 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.8.log 2023-07-20T01:45:30.790 INFO:teuthology.orchestra.run.smithi086.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T01:45:30.790 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.f.log 2023-07-20T01:45:30.790 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.c.log 2023-07-20T01:45:30.790 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.8.log: /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.f.log: /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.c.log: 91.2% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mgr.z.log.gz 2023-07-20T01:45:30.793 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.i.log 2023-07-20T01:45:30.793 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mon.c.log 2023-07-20T01:45:30.793 INFO:teuthology.orchestra.run.smithi086.stderr: 93.7% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.f.log.gz 2023-07-20T01:45:30.793 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mon.c.log: 93.2% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.c.log.gz 2023-07-20T01:45:30.795 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.3.log: /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.d.log: 94.8% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.d.log.gz 2023-07-20T01:45:30.796 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mgr.x.log 2023-07-20T01:45:30.796 INFO:teuthology.orchestra.run.smithi003.stderr: 94.0% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.g.log.gz 2023-07-20T01:45:30.796 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.9.log 2023-07-20T01:45:30.796 INFO:teuthology.orchestra.run.smithi003.stderr: 93.9% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.a.log.gz 2023-07-20T01:45:30.796 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T01:45:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.9.log: /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-volume.log: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T01:45:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.j.log 2023-07-20T01:45:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.j.log: 94.8% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.j.log.gz 2023-07-20T01:45:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mon.a.log 2023-07-20T01:45:30.797 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mon.a.log: 91.5% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T01:45:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T01:45:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:: No such file or directory 2023-07-20T01:45:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.10.log 2023-07-20T01:45:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.h.log 2023-07-20T01:45:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.e.log 2023-07-20T01:45:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.4.log 2023-07-20T01:45:30.799 INFO:teuthology.orchestra.run.smithi043.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T01:45:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.1.log 2023-07-20T01:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.k.log 2023-07-20T01:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.4.log: /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T01:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.k.log: /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.7.log 2023-07-20T01:45:30.801 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T01:45:30.801 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.l.log 2023-07-20T01:45:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.7.log: 94.0% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.k.log.gz 2023-07-20T01:45:30.802 INFO:teuthology.orchestra.run.smithi043.stderr: 93.3% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.h.log.gz 2023-07-20T01:45:30.802 INFO:teuthology.orchestra.run.smithi043.stderr: 93.8% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.e.log.gz 2023-07-20T01:45:30.802 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mon.b.log 2023-07-20T01:45:30.802 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.b.log 2023-07-20T01:45:30.802 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mon.b.log: /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mgr.y.log 2023-07-20T01:45:30.802 INFO:teuthology.orchestra.run.smithi043.stderr: 91.4% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T01:45:30.802 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mgr.y.log: 91.1% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mgr.y.log.gz 2023-07-20T01:45:30.803 INFO:teuthology.orchestra.run.smithi043.stderr: 93.4% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.b.log.gz 2023-07-20T01:45:30.803 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-volume.log: /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.l.log: 94.3% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.l.log.gz 2023-07-20T01:45:30.804 INFO:teuthology.orchestra.run.smithi086.stderr: 91.5% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T01:45:30.819 INFO:teuthology.orchestra.run.smithi086.stderr: 93.4% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mds.i.log.gz 2023-07-20T01:45:30.842 INFO:teuthology.orchestra.run.smithi086.stderr: 93.3% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.11.log.gz 2023-07-20T01:45:30.859 INFO:teuthology.orchestra.run.smithi003.stderr: 92.8% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.6.log.gz 2023-07-20T01:45:30.860 INFO:teuthology.orchestra.run.smithi003.stderr: 93.1% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.0.log.gz 2023-07-20T01:45:30.890 INFO:teuthology.orchestra.run.smithi003.stderr: 92.8% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.9.log.gz 2023-07-20T01:45:30.892 INFO:teuthology.orchestra.run.smithi043.stderr: 92.8% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.7.log.gz 2023-07-20T01:45:30.896 INFO:teuthology.orchestra.run.smithi003.stderr: 92.8% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.3.log.gz 2023-07-20T01:45:30.897 INFO:teuthology.orchestra.run.smithi086.stderr: 92.9% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.8.log.gz 2023-07-20T01:45:30.898 INFO:teuthology.orchestra.run.smithi043.stderr: 92.0% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mon.b.log.gz 2023-07-20T01:45:30.902 INFO:teuthology.orchestra.run.smithi086.stderr: 92.7% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.5.log.gz 2023-07-20T01:45:30.905 INFO:teuthology.orchestra.run.smithi003.stderr: 88.8% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mgr.x.log.gz 2023-07-20T01:45:30.906 INFO:teuthology.orchestra.run.smithi043.stderr: 92.7% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.4.log.gz 2023-07-20T01:45:30.909 INFO:teuthology.orchestra.run.smithi086.stderr: 92.9% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.2.log.gz 2023-07-20T01:45:30.909 INFO:teuthology.orchestra.run.smithi043.stderr: 93.3% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.1.log.gz 2023-07-20T01:45:30.913 INFO:teuthology.orchestra.run.smithi043.stderr: 93.0% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-osd.10.log.gz 2023-07-20T01:45:30.915 INFO:teuthology.orchestra.run.smithi043.stderr: 2023-07-20T01:45:30.915 INFO:teuthology.orchestra.run.smithi043.stderr:real 0m0.145s 2023-07-20T01:45:30.915 INFO:teuthology.orchestra.run.smithi043.stderr:user 0m0.613s 2023-07-20T01:45:30.915 INFO:teuthology.orchestra.run.smithi043.stderr:sys 0m0.037s 2023-07-20T01:45:30.927 INFO:teuthology.orchestra.run.smithi086.stderr: 92.0% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mon.c.log.gz 2023-07-20T01:45:30.928 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-07-20T01:45:30.929 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m0.160s 2023-07-20T01:45:30.929 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m0.611s 2023-07-20T01:45:30.929 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.045s 2023-07-20T01:45:31.122 INFO:teuthology.orchestra.run.smithi003.stderr: 91.0% -- replaced with /var/log/ceph/9b4fd2fc-269d-11ee-9b34-001a4aab830c/ceph-mon.a.log.gz 2023-07-20T01:45:31.124 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-07-20T01:45:31.124 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m0.357s 2023-07-20T01:45:31.124 INFO:teuthology.orchestra.run.smithi003.stderr:user 0m0.838s 2023-07-20T01:45:31.124 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m0.050s 2023-07-20T01:45:31.124 INFO:tasks.cephadm:Archiving logs... 2023-07-20T01:45:31.125 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251/remote/smithi003/log 2023-07-20T01:45:31.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T01:45:31.302 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251/remote/smithi043/log 2023-07-20T01:45:31.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T01:45:31.406 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251/remote/smithi086/log 2023-07-20T01:45:31.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T01:45:31.508 INFO:tasks.cephadm:Removing cluster... 2023-07-20T01:45:31.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c --force 2023-07-20T01:45:31.803 INFO:teuthology.orchestra.run.smithi003.stdout:Deleting cluster with fsid: 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:45:32.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c --force 2023-07-20T01:45:32.574 INFO:teuthology.orchestra.run.smithi043.stdout:Deleting cluster with fsid: 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:45:33.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9b4fd2fc-269d-11ee-9b34-001a4aab830c --force 2023-07-20T01:45:33.335 INFO:teuthology.orchestra.run.smithi086.stdout:Deleting cluster with fsid: 9b4fd2fc-269d-11ee-9b34-001a4aab830c 2023-07-20T01:45:33.790 INFO:tasks.cephadm:Removing cephadm ... 2023-07-20T01:45:33.790 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T01:45:33.813 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T01:45:33.836 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T01:45:33.853 INFO:tasks.cephadm:Teardown complete 2023-07-20T01:45:33.854 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T01:45:33.934 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f08713bff0734a2f97c42d077a05d9f5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T01:45:33.941 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-20T01:45:33.953 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-20T01:45:34.000 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-20T01:45:34.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T01:45:34.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T01:45:34.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T01:45:34.137 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-20T01:45:34.148 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-20T01:45:34.148 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:45:34.151 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:45:34.154 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:45:34.167 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2023-07-20T01:45:34.170 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2023-07-20T01:45:34.172 INFO:teuthology.orchestra.run.smithi043.stderr:bash: ntpq: command not found 2023-07-20T01:45:34.173 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:45:34.173 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-07-20T01:45:34.174 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 -1745us[-1744us] +/- 82ms 2023-07-20T01:45:34.174 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -2496us[-2495us] +/- 78ms 2023-07-20T01:45:34.174 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +4820us[+4822us] +/- 90ms 2023-07-20T01:45:34.174 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:45:34.175 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:45:34.176 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2023-07-20T01:45:34.176 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 -1632us[-1632us] +/- 82ms 2023-07-20T01:45:34.177 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -2375us[-2386us] +/- 78ms 2023-07-20T01:45:34.177 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +4928us[+4928us] +/- 90ms 2023-07-20T01:45:34.177 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:45:34.178 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:45:34.179 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2023-07-20T01:45:34.179 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 -1712us[-1708us] +/- 82ms 2023-07-20T01:45:34.179 INFO:teuthology.orchestra.run.smithi043.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -2411us[-2407us] +/- 78ms 2023-07-20T01:45:34.179 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +4855us[+4855us] +/- 90ms 2023-07-20T01:45:34.179 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:45:34.180 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-20T01:45:34.191 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-20T01:45:34.192 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-20T01:45:34.218 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T01:45:34.320 DEBUG:teuthology.orchestra.run.smithi043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T01:45:34.424 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T01:45:34.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:45:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:45:34.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:45:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:45:34.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:45:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:45:34.619 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-20T01:45:34.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-20T01:45:34.660 INFO:teuthology.task.internal:Duration was 1414.271284 seconds 2023-07-20T01:45:34.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-20T01:45:34.694 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-20T01:45:34.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T01:45:34.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T01:45:34.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T01:45:34.745 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:45:34.746 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:45:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:45:35.112 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-20T01:45:35.112 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2023-07-20T01:45:35.112 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T01:45:35.137 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi043.front.sepia.ceph.com 2023-07-20T01:45:35.137 DEBUG:teuthology.orchestra.run.smithi043:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T01:45:35.162 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2023-07-20T01:45:35.163 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T01:45:35.185 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-20T01:45:35.186 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T01:45:35.188 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T01:45:35.204 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T01:45:35.272 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-20T01:45:35.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T01:45:35.275 DEBUG:teuthology.orchestra.run.smithi043:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T01:45:35.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T01:45:35.370 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-20T01:45:35.382 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-20T01:45:35.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T01:45:35.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T01:45:35.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T01:45:35.465 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-20T01:45:35.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:45:35.478 DEBUG:teuthology.orchestra.run.smithi043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:45:35.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:45:35.502 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2023-07-20T01:45:35.505 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = core 2023-07-20T01:45:35.531 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2023-07-20T01:45:35.552 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:45:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:45:35.603 DEBUG:teuthology.orchestra.run.smithi043:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:45:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:45:35.620 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:45:35.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:45:35.638 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-20T01:45:35.649 INFO:teuthology.task.internal:Transferring archived files... 2023-07-20T01:45:35.651 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251/remote/smithi003 2023-07-20T01:45:35.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T01:45:35.696 DEBUG:teuthology.misc:Transferring archived files from smithi043:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251/remote/smithi043 2023-07-20T01:45:35.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T01:45:35.734 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251/remote/smithi086 2023-07-20T01:45:35.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T01:45:35.776 INFO:teuthology.task.internal:Removing archive directory... 2023-07-20T01:45:35.777 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T01:45:35.779 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T01:45:35.781 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T01:45:35.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-20T01:45:35.835 INFO:teuthology.task.internal:Not uploading archives. 2023-07-20T01:45:35.835 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-20T01:45:35.861 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-20T01:45:35.862 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T01:45:35.864 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T01:45:35.866 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T01:45:35.878 INFO:teuthology.orchestra.run.smithi003.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 01:45 /home/ubuntu/cephtest 2023-07-20T01:45:35.880 INFO:teuthology.orchestra.run.smithi043.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 01:45 /home/ubuntu/cephtest 2023-07-20T01:45:35.882 INFO:teuthology.orchestra.run.smithi086.stdout: 262181 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 01:45 /home/ubuntu/cephtest 2023-07-20T01:45:35.883 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-20T01:45:35.895 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-20T01:45:35.971 INFO:teuthology.nuke:Checking targets against current locks 2023-07-20T01:45:35.998 DEBUG:teuthology.nuke:shortname: smithi003 2023-07-20T01:45:35.999 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T01:45:36.025 DEBUG:teuthology.nuke:shortname: smithi043 2023-07-20T01:45:36.026 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T01:45:36.030 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 01:16:48.200735', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T01:45:36.052 DEBUG:teuthology.nuke:shortname: smithi086 2023-07-20T01:45:36.053 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T01:45:36.060 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 01:16:48.197624', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T01:45:36.079 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 01:16:48.199280', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T01:45:36.090 INFO:teuthology.orchestra.console:Power off smithi003 2023-07-20T01:45:36.091 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T01:45:36.115 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:45:36.140 INFO:teuthology.orchestra.console:Power off smithi043 2023-07-20T01:45:36.141 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T01:45:36.166 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:45:36.188 INFO:teuthology.orchestra.console:Power off smithi086 2023-07-20T01:45:36.189 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T01:45:36.213 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:45:40.154 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:45:40.202 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:45:40.239 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:45:44.272 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:45:44.317 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:45:44.352 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:45:44.484 INFO:teuthology.orchestra.console:Power off for smithi003 completed 2023-07-20T01:45:44.531 INFO:teuthology.orchestra.console:Power off for smithi043 completed 2023-07-20T01:45:48.472 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:45:48.686 INFO:teuthology.orchestra.console:Power off for smithi086 completed 2023-07-20T01:45:48.797 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} duration: 1414.2712836265564 failure_reason: 'Command failed on smithi003 with status 2: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f08713bff0734a2f97c42d077a05d9f5 status: fail success: false 2023-07-20T01:45:48.798 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T01:45:48.886 INFO:teuthology.run:FAIL